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Executive Summary 



This is the final report of the National Evaluation of Student Support Services (SSS). SSS is 
one of eight federally funded grant programs that are administered as part of the Federal TRIO Programs 
within the U.S. Department of Education (ED). The SSS program, in particular, focuses on students 
while they are enrolled in college. In general, SSS provides the most services to first-year college 
students, though it does also provide services in later years. 

The purpose of the study was to estimate the effects of SSS on the outcomes of the student 
participants. The full report discusses five academic outcomes. For brevity, this summary focuses only 
on the key outcomes: retention in college, transfers from two-year to four-year institutions, and degree 
completion. 

Overview of SSS 

The purposes of the SSS program, as stated in the Higher Education Act of 1965, as 
reauthorized by the Higher Education Opportunity Act of 2008, are to (1) increase college retention and 
graduation rates for eligible students, (2) increase the transfer rates of eligible students from two-year to 
four-year institutions, and (3) foster an institutional climate supportive of success for low-income and 
first-generation college students and individuals with disabilities. Two-thirds of the students served by an 
SSS project must be low-income (defined as at or below 150 percent of the poverty level) and first- 
generation college students or students with disabilities. The other third must be low-income or first- 
generation college students. One-third of the disabled students also must be low income. 

SSS projects have great latitude to custom-design their services to fit the particular needs of 
their student population. All SSS projects provide academic advising as one of their services (although 
there is great variation in the amount students received), but the projects differ greatly with respect to 
offering other services to SSS students such as tutoring, labs, workshops, special instructional courses, 
and services specifically for students with disabilities. As a rule, SSS students are in full control of 
determining both the types and the amounts of services they receive, as long as the services are offered by 
the institutions and the students qualify for them (e.g., a student must have a disability in order to receive 
special services for individuals with disabilities). 

Main Findings 

The study used a variety of statistical methodologies to estimate the effects of participating 
in the SSS program as a college freshman (the only year for which it was possible to collect program 
participation data) and also the more general effect of receiving supplemental services during college. 

The major finding is that analytic models that account for differences in service levels 
generally showed positive and statistically significant effects. Participation in SSS projects as measured 
by the amount of services received during the freshman year is associated with moderate increases on the 
key measures of college retention and degree completion but neither increases nor decreases student 
transfers from two-year to four-year institutions and neither increases nor decreases the outcomes on 
some of the key measures in the HLM models. Although these models controlled for student 
demographics and, whenever possible, prior achievement, one limitation of this model is the potential 
selection bias of participants who received more services. 

Models that measure supplemental services regardless of whether they were offered by the 
SSS project or were offered by some other service provider on campus are associated with positive and 
statistically significant effects on all outcome measures of retention, transfers from two-year to four-year 
institutions, and degree completion. 
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In addition this report includes analyses that simply consider whether or not the student was 
classified as being in SSS as a college freshman, although this comparison was considered of limited 
validity given the structure of SSS grants. A major limitation of this analysis is that it does not account 
for the level of service received by SSS participants; nor does it account for the fact that comparison 
students may have received similar services that were not funded by the federal SSS program. This 
measure did not show any effect from participating in SSS as a college freshman. 

Study Design 

This report focuses on a statistical comparison of SSS and non-SSS students in order to 
estimate the effects of SSS. The study was designed as a longitudinal study, in which the academic 
progress of 5,800 freshman students in 1991-92 was tracked for six years. Half of the students were 
participants in SSS, and the other half were statistically chosen using propensity models to have similar 
characteristics to the SSS participants. Both quantitative and qualitative data were collected. It should be 
noted that this study utilizes data from the 1990s and may not capture the full nature of the SSS program 
as it operates today. 

The sample was chosen in several steps. First, a nationally representative, stratified random 
sample of 200 SSS projects was surveyed in 1991-92. Second, 30 of these projects were randomly 
subsampled for site visits and other data collection activities, and 20 higher education institutions without 
SSS grants were selected to match the SSS sites, based on institutional characteristics. Three of the 
institutions later dropped out of the study. The 47 remaining institutions provided basic information on 
student characteristics, and the SSS projects also provided participant service records on each service 
contact through SSS for the sampled students, including the types of service, length in minutes, date, and 
number of students receiving the specific services. Student surveys were conducted at three different time 
points: 1991-92, 1994-95, and 1997-98. Student transcripts were collected from all institutions the 
students attended (not just the 47 original institutions) at the end of the first, third, and sixth years. 

Methodology 

This study had a quasi-experimental design. Regression models and propensity scores were 
used to select a group of comparison students that matched the SSS students as closely as possible on a 
variety of student characteristics, and both the SSS and comparison students were monitored through 
student surveys to determine what other supplemental services they received. Although the comparison 
students were highly disadvantaged, the SSS participants were even more so, and additional statistical 
adjustments were required in the analysis to allow appropriate comparisons between the two groups. 
Exposure to SSS was not randomized because of the way in which SSS is structured. By design, SSS 
students may receive supplemental services outside of SSS, and any attempt to deny services to needy 
students would raise ethical questions, threaten people’s support of the institution, and potentially conflict 
with the institution’s mission. 

The lack of a uniform SSS experience, with considerable variation even within each 
institution, combined with the receipt of equivalent services outside of SSS, made it difficult to design 
statistical models that properly described students’ experience. Rather than choosing a single 
methodology, therefore, this study used multiple approaches. All models included separate measures of 
SSS services for first-year students, supplemental services received outside of SSS or received after the 
first year, and measures of student and school characteristics. The models differed in the statistical 
techniques that were used, in the ways that SSS and other supplemental services were measured, and in 
the use of propensity scores. 

■ Some models used standard multivariate regression analysis while others use 
hierarchical linear modeling (HLM). 
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■ Some models treated SSS participation as dichotomous (i.e., either a student was in 
SSS or he or she was not), while other models treated SSS participation as a collection 
of nine separate services, with each service measured separately in terms of the 
number of hours of participation by each student. 

■ Some models adjusted for differences among students solely through individual 
measures of student characteristics (such as academic strength, background, and 
attitudes), while other models also included propensity measures that are designed to 
estimate students’ probability of receiving services. 

This report includes all of the above approaches so that readers can see the implications of 
the various methodological choices. In principle, HLM provides better estimates than regressions 
because it is better able to handle the clustering of students within institutions. However, HLM did not 
always converge to a solution when using logistic models (for limited dependent variables), thus requiring 
the use of more standard regression models within HLM. Dichotomous measures of services avoid the 
difficulty that students’ levels of participation may be related to other factors (such as neediness or 
motivation) that might have confounding relationships with student outcomes. However, due to the high 
correlation between SSS participation and the receipt of other supplemental services (91 percent of the 
SSS students received services outside of SSS), dichotomous measures primarily are useful for examining 
supplemental services in general. Only the models with continuous measures appear able to discriminate 
between SSS and other services. Propensity scores usually serve as a substitute for individually fitting 
each regression equation based on student characteristics, and they should be less necessary in these 
models that were individually fitted. In practice, the propensity scores often were statistically 
insignificant, and estimated SSS effects were increased when the statistically insignificant propensity 
scores were dropped. Thus, it is questionable whether they provided useful information, and they may 
sometimes have had confounding effects because of their interrelation with the measures of SSS services. 

The regression estimates are presented here in two different formats: as regression 
coefficients to aid researchers to understand the structure of the models and to replicate the results, and as 
impact estimates to aid policymakers in applying the findings, because the regression estimates often do 
not translate easily into effects. (One difficulty is that for some models, one must know the amount of 
services that students received in order to interpret the regression coefficients; another is that when 
logistic regression is used, neither the regression coefficients nor the log odds ratios are easy to interpret.) 

Normally, estimates of program effects would be presented by indicating both what students 
would have achieved if they did not participate in SSS and what they did achieve after participating in 
SSS. Unfortunately, because the comparison group was more advantaged than the SSS participants, and 
because students in the comparison group often also received supplemental services, there is no pure 
measure of what outcomes would be expected if students had not participated in SSS. There are actual 
measures of final student outcomes after six years, but the alternative outcomes if students had not 
participated in SSS could only be estimated through statistical models. Moreover, because this report 
uses multiple models as a tool for examining the implications of the methodological choices involved, 
there are multiple estimates of the effects of SSS from which to choose. The results across the various 
models are often highly consistent but not identical. Therefore this report presents only the estimated 
effects of the services along with the final measured outcomes. Readers can calculate what students 
would have achieved without SSS by first deciding which model appears most trustworthy, and then 
subtracting the estimated effects from the final outcomes. 

This study was longitudinal in the sense that the selected students were tracked for six years 
with multiple data collections in order to determine their experiences during those six years and their 
academic outcomes. However, the data analysis has been largely cross-sectional, with the multiple data 
points all included in a single regression or HLM model rather seeking to model changes in the students 
from one time to another. An exception is that in the previously published analysis of student outcomes 
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after three years, structural equations models were used to model the interactions between SSS 
participation, first-year college GPAs, and college retention. That is, one might anticipate that students’ 
academic successes in the first year, as measured by their GPAs, would influence their retention in 
college, but their academic successes could be influenced both by their own characteristics and by 
assistance they received from SSS or other sources. Thus, if first-year college GPAs are used to predict 
retention, a regression equation would tend to underestimate the effect of SSS and other supplemental 
services by only estimating that portion of the effect that was independent of students' GPAs, while 
ignoring the indirect contribution of supplemental services through their effects on GPAs. For that 
reason, structural equations models were used to separate students' GPAs into a latent GPA that would be 
expected without any receipt of supplemental services and the additional increment added through 
supplemental services; the latent GPA, rather than the actual GPA, was then used to predict retention so 
the model could estimate the full effects of SSS. 

The structural equations model confirmed that SSS did indirectly influence retention through 
its effect on GPAs, and that the effect of SSS on retention was underestimated unless this relationship was 
properly modeled. This analysis helps to demonstrate the importance of structural equations and path 
analytic models in longitudinal studies. However, the same analysis also revealed that when a second 
measure of students’ academic strengths (high school GPAs) was substituted for college GPAs (because 
high school GPAs would not have been influenced by SSS), essentially the same estimates of the effect of 
SSS were produced. If the purpose of this study was to develop a full theoretical model of what happens 
to students while in college, then it would be important to model the different paths through which SSS 
influences students. However, because the study goal was simply to estimate the total effects of SSS, and 
not to estimate how that effect was distributed among direct and indirect paths, this report uses surrogate 
measures such as high school GPAs that arc independent of SSS rather than a structural equations 
approach. This choice avoided some technical complexities (such as imputation of missing data and the 
use of logistic regression within structural equations models) that would have been introduced with a 
structural equations approach. 

Findings From Earlier Years of This Study 

This report focuses on student outcomes six years after freshmen began participating in SSS. 
As part of the same longitudinal study, earlier reports discussed the effects of SSS on student outcomes 
after one year and after three years. The outcomes that were examined earlier were largely the same 
outcomes as those discussed here (effects on grade point averages, credits earned, and retention, though 
over shorter time frames), except that the one- or three-year time periods were too short to examine the 
effects on degree completion and on transfers from two-year to four-year institutions. 

SSS showed a small but positive and statistically significant effect on students’ GPAs, 
number of semester credits earned, and retention. The greatest effect generally occurred during the first 
year, when the most SSS services were received, but some SSS services received in the first year showed 
persisting effects in later years, and some services received in later years (not necessarily through SSS) 
also showed positive and statistically significant effects. The size of the effect depended on the degree to 
which students participated in SSS, with greater levels of participation resulting in a greater effect. The 
estimated effects of SSS also varied based on which particular services each student received and the 
structure of the SSS projects. 

The average effect was small because most students received only a modest amount of 
services. Nine percent of students had only one service in their freshman year. The mean number of 
hours of services received in the first year was 32, and the median was 14. The mean for nonfreshmen 
was 15 hours, and the median was six. SSS projects appeared successful in targeting those students who 
were most disadvantaged from among the overall student population. In comparison with the national 
averages for college freshmen, SSS participants tended to be older, to be members of a minority group, to 
have had lower prior academic achievement, and to have dependent children. For additional highlights 
from the previous study, see page 7 in the Final Report. 
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Limitations 



Except for services received during the first year, this study was not able to distinguish 
between services received through SSS and services received through other means. The reason is that to 
limit the burden on participating institutions and to obtain their cooperation, the data on later-year 
services were collected only through student questionnaires, and students were not expected to know the 
funding source of the services that they received (especially since SSS programs often were renamed). 

The third-year report found that effects from first-year services were greater on first-year outcomes than 
on later outcomes, so it is consistent that six-year outcomes show a stronger relationship to later-year 
services than to first-year services. Thus, though some findings specifically concerning SSS do appear in 
this report, many of the findings are more general in nature and apply to all supplemental services 
regardless of the source. In one sense, this is not a serious limitation; there is no specific reason that, say, 
peer tutoring would have a different level of effectiveness when provided through SSS than through other 
means. Findings about the effects of supplemental services can be interpreted as providing information 
about the effects of SSS when it supplies those same supplemental services. Also, participation in SSS 
was associated with receiving higher levels of services (whether through SSS or through other sources), 
suggesting that some of the improved outcomes from later-year services might be associated with SSS 
either because the services were provided by SSS or because SSS encouraged students to make fuller use 
of available services. However, the specific amount of the contribution provided by SSS in comparison 
to other sources of supplemental services generally cannot be determined. 

In the case of services received after the first year, any student who received such services 
necessarily persisted beyond the first year, so that the presence of such services is partially correlated with 
academic success. This could lead to overstating the importance of later-year services. The problem is 
most likely to appear when dichotomous measures of SSS participation are used, while the greater amount 
of variation in the predictors when multiple and continuous measures are used helps to lessen the 
difficulty. To adjust for this issue, all of the models include a variable indicating whether the student 
attended higher education at any time during the five years after 1991-92. That way the changes in 
outcomes that are associated with persistence are measured through this variable, while the measure of the 
receipt of supplemental services will only capture any additional changes in student outcomes that are 
associated with the receipt of supplemental services. However, the addition of this variable may lead to 
underestimates of the effects of supplemental services (whether through SSS or through other means). 
Because persistence may be partially due to the receipt of supplemental services, the estimates for 
supplemental services may show only that effect that is independent of this measure of persistence, and 
not the effect that is related to the impact of supplemental services on persistence. 

Some SSS services were received by only a small number of students, affecting the 
likelihood of finding statistically significant results. Of the 5,800 students originally in the study, 5,055 
ultimately had sufficient data for inclusion in the six-year outcomes analysis, with 2,380 being SSS 
participants. Depending on the service, a range from 63 to 1,892 students received each service. The 
number of students was particularly an issue for analyses that were limited to students at two-year 
institutions. Among students at two-year institutions, between eight and 415 students received each 
service. The small number of students receiving some services may also have made it more difficult to 
measure the effects of SSS services than of general supplemental services (i.e., using measures that do not 
distinguish between SSS and non-SSS services), because roughly twice the number of students would 
have received services when using these more general measures. 

Study Findings: Examination of Student Outcomes After Six Years 

The single most consistent finding is that the receipt of supplemental services was correlated 
with improved student academic outcomes. This finding was consistent across all measures of academic 
outcomes and all types of statistical models (exhibit E-l). The findings for first-year SSS services in 
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particular were also largely consistent and positive, except that the models using dichotomous measures 
did not show any effect from SSS. The poor performance of the dichotomous measures was anticipated 
because of the difficulty of differentiating between SSS and non-SSS services in those models, and 
appears to be a methodological artifact. The continuous measures of SSS participation showed positive 
and statistically significant effects for each of the outcome measures except transfers from two-year to 
four-year institutions and selected models using HLM. Participation in SSS was also associated with 
receiving a higher level of supplemental services, including both those services specifically designed for 
SSS students and supplemental services in general. 

A second finding is that supplemental services continued to be important after the freshman 
year. In fact, the later-year services appear to show a stronger relationship to long-term outcomes than 
first-year services. SSS programs typically give the greatest emphasis to providing services for freshmen, 
though they include services for later years as well. Based on this finding, some SSS programs may wish 
to reexamine their distribution of services to be sure that some program is in place to provide services in 
later years. 

Exhibit E-l. Estimated improvement in academic outcomes associated with receiving first-year 



SSS services and with receiving supplemental services from any source and in any of 
the six academic years, using six analytic approaches: Six-year outcomes after 
freshman entry in 1991-92 



Methodology 


Six-year 


Credits 


Retention or degree 
completion 
(combined) 


Transfers 
from two-year 


Degree attainment 


GPA 


earned 


Same 

institution 


Any 

institution 


to four-year 
institutions 


Bachelor's 
degree or 
higher 


Associate’s 
degree or 
higher 


Observed six-year outcome 


2.34 


75.49 


34% 


63% 


19% 


38% 


49% 


Dichotomous measures of 
services 
Regressions 

SSS services only 


(Increase) 

0.06 




(Percentage point increase) 




All supplemental services 


0.23 


21.97 


12% 


16% 


11% 


10% 


8% 


HLM 

SSS services only 
















All supplemental services 


0.14 


16.99 


12% 


18% 


13% 


10% 


8% 


Continuous measures of 
services 

No propensity scores 
Regressions 

SSS services only 


0.12 


2.60 


4% 


5% 




6% 


3% 


All supplemental services ... 


0.25 


18.77 


13% 


21% 


7% 


14% 


11% 


HLM 

SSS services only 


0.04 


3.19 


5% 


9% 




2% 


1% 


All supplemental services ... 


0.17 


19.17 


15% 


24% 


10% 


13% 


12% 


With propensity scores 
Regressions 

SSS services only 


0.14 


2.39 


7% 


8% 




5% 




All supplemental services ... 


0.29 


18.48 


16% 


23% 


8% 


14% 


9% 


HLM 

SSS services only 


0.02 




6% 


4% 








All supplemental services ... 


0.16 


16.27 


17% 


19% 


10% 


11% 


11% 



-- Not statistically significant. Statistical significance is based on p-value of 0.05. 

NOTE: The six-year outcome is the mean outcome for all applicable SSS participants who were in the study, as measured through their 



transcripts and self-reports. The predicted outcome if students did not receive services can be calculated for any of the six analytic approaches by 
subtracting the percentage point improvement from the observed six-year outcome. For example, the mean retention rate at the same institution 
among all SSS participants in the study was 34 percent, and using the HLM model with propensity scores the mean retention rate is estimated at 
17 percent (34 percent minus 17 percent) if they had not received any supplemental services. 
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Third, a few SSS services appeared to stand out by being related to improved student 
outcomes: home -based programs, blended programs, peer tutoring, labs, workshops, and services for 
students with disabilities. However, some additional types of services also were related to improved 
student outcomes, though they were not necessarily SSS services: counseling, field trips or cultural 
enrichment, referrals to outside resources, services for those with limited English ability, college re- 
entrance counseling, and recent contacts with support services. Also, there is some evidence that what 
may be most important is that students receive an appropriate “package” of services, and some individual 
measures may not have shown independent benefits because their benefits were included in measures of 
whether the students had received packages of services (e.g., through home-based or blended programs). 

Finally, the stronger findings for supplemental services in general when compared with first- 
year SSS services in particular should not be interpreted as indicating that services outside of SSS were 
superior to SSS services. Instead, two reasons seem the best explanations for the differences: (1) later- 
year services may be more effective than first-year services when seeking to predict long-term outcomes 
(i.e., because they arc more contemporaneous with the long-term outcomes); and (2) some of the effects 
of SSS services may be captured through the other measures of supplemental services, so that the SSS 
effect is underestimated. The latter explanation seems likely both because some of the later-year services 
were likely to have been provided by SSS services, and because of the difficulty of differentiating SSS 
from non-SSS services when using dichotomous measures of participation. Also, the use of propensity 
measures may sometimes also have led to underestimating the effects of SSS. Often the propensity scores 
were insignificant, suggesting that they were not important parts of the models, and if the insignificant 
measures were dropped from the models, the estimated effect of SSS was increased. 
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1. Introduction 



Student Support Services (SSS) is one of eight federally funded grant programs that are 
administered as part of the Federal TRIO Programs within the U.S. Department of Education (ED). 1 All 
eight programs are designed to help economically disadvantaged and first-generation college students 
achieve success at the postsecondary level. Some programs work with students at the middle school or 
high school level to help them complete high school and prepare for postsecondary education. Other 
programs help students while they are attending postsecondary institutions. The SSS program, in 
particular, focuses on students while they are enrolled in college and is designed to help them complete 
their postsecondary degrees or certificates, to transfer to four-year institutions if they start at two-year 
schools, and to enroll in graduate school. Students may start participating in SSS immediately upon 
enrollment in postsecondary education or at a later stage. In general, SSS provides the most services to 
first-year college students, though it does provide services in later years as well. 

SSS has grown from 121 projects and approximately 30,000 participants in 1970-71 to 947 
projects and 198,940 participants in 2007-08. Its funding has increased from $10 million to $264 million 
($52 million in 1970 dollars). Participating institutions comprise roughly 22 percent of the four-year and 
two-year colleges and universities serving freshmen in the United States. SSS is the second largest of the 
Federal TRIO Programs in terms of the number of students served (after Talent Search), and the second 
largest in terms of funding (after Upward Bound). 

This report presents the final results of a longitudinal study of students who began 
participating in SSS during the 1991-92 academic year. This study was mandated by Congress to 
“examine the effectiveness of current programs and to identify program improvements” (P.L. 101-166). 
Earlier results were presented in an implementation study report, completed in 1994, which described the 
characteristics of the SSS program and the SSS students at the institutions participating in the study, and 
in a third-year update, completed in 1997, which provided additional information on the implementation 
of SSS as well as an analysis of the effects of SSS on participating students with regard to their grade 
point averages (GPAs), the number of credits they earned, and their retention in higher education. This 
study updates the previous analysis by examining the effects of SSS after six years. It continues the 
analysis of GPAs, total credits earned, and retention from the earlier study, but also expands the analysis 
by examining degree completion and transfers to four-year institutions. These last two measures could 
not have been meaningfully examined in earlier years because students often had not had sufficient time 
to complete their education. The measures are much more meaningful now when applied to six years of 
time, although even that amount of time is not always enough for all students to have finished their 
education. 



This evaluation was conducted as a longitudinal study, in which both SSS participants and a 
comparison group of nonparticipants were tracked for six years following their entry into college. The 
students were selected by first selecting 200 SSS projects through a nationally representative random 
sample, subsampling 30 of those institutions and 20 comparable institutions without SSS projects for site 
visits, and selecting groups of SSS participants and nonparticipants from those institutions. Of these 50 
institutions, 47 ultimately participated for all phases of the study. From these sites and their participating 



1 The eight federal programs are collectively known as the Federal TRIO Programs in the Higher Education Act of 1 965, as 
reauthorized by the Higher Education Opportunity Act of 2008. The other TRIO programs are Upward Bound, Talent Search, 
the Educational Opportunity Centers (EOC), Staff Training, the McNair Fellowships. Dissemination Partnership Program, and 
Upward Bound Math-Science. 
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students, the following data were collected: student information files containing basic information on 
student characteristics; participant service records documenting each student’s contact with SSS during 
the first year; student surveys conducted in 1991-92, 1993-94, and 1996-97; and student transcripts for 
students’ six years of postsecondary education starting in 1991-92. Additional information about the 
research methodology is provided in chapter 2. 

In many ways, this report builds upon the methodology and results of the third-year follow- 
up study. Some of the particularly important findings of that report that relate to the design of the study 
are listed below. Additional details about past findings are presented later in this chapter. 

■ The 2,900 non-SSS study participants tended to be relatively advantaged in comparison 
to the SSS students, in part due to a lack of sufficient information to identify similar 
students and in part due to the fact that some institutions did not have students who were 
comparable in their disadvantages to the SSS participants. As a result, multivariate 
analysis is used in this report (as it was previously) to statistically adjust for the 
differences between these two groups. 

■ SSS students were found to vary greatly in the types and amounts of services they 
received through SSS, both across SSS projects and within individual SSS projects. 

This report therefore uses a variety of methodological approaches to look at the 
importance of these variations. 

■ Student demographic characteristics were important, with blacks, Hispanics, and males 
often showing poorer academic outcomes on average. Thus, these types of variables 
were also examined in this report and retained when they were statistically significant. 

■ Similarly, the institution type (two-year or four-year) was associated with student 
outcomes and is retained here for use in the updated multivariate models. 

This report makes selected use of the statistics and findings from the earlier reports but does 
not attempt to include all of them. Readers desiring additional information are particularly advised to 
examine the third-year follow-up report, which includes substantial information about the SSS program 
and the characteristics of SSS participants. 

Organization of This Report 

Because of the way that SSS programs are designed, with programs varying greatly from one 
institution to another, and students varying in the degree to which they participate, the design and analysis 
of the evaluation raised some complex methodological issues. The methodological choices potentially 
may have an important effect on the research findings, and researchers may legitimately disagree on the 
best methodological approach. Therefore, the methodological decision points are given special 
prominence in this report, and alternative methodologies are compared so that readers can determine the 
importance of the choices and give special attention to the approaches that they prefer. This report is 
structured to facilitate such comparisons. 

This chapter provides an overview of the SSS program and its students. Chapter 2 presents 
an overview of the study design and discusses the key methodological issues and their implications for 
the study. The remainder of the report progressively examines the data using each approach and 
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discusses the findings in the context of each approach. Chapter 3 starts the analysis using the simplest 
statistical models, in which dichotomous measures of SSS and non-SSS participation are used. Some 
advantages of this approach are that it provides a single summary measure of the effects of SSS, and that 
differences in participation levels among SSS participants are ignored (because those differences might be 
indicators of differences in neediness or motivation, which also might be related to student outcomes). 
Chapter 3 includes both regression analysis and HLM so that readers can compare the two methodologies. 
Chapter 4 presents models in which SSS participation is measured through nine separate variables for 
each of the nine SSS services listed in exhibit 1-1, plus two organizational characteristics of SSS that 
were noted in the site visits. Some potential advantages of this approach are an increased ability to 
differentiate between SSS and non-SSS services (in order to measure the full effect of SSS), a better 
capacity to distinguish among SSS students to determine whether some students are helped more than 
others (e.g., based on their levels of participation in SSS), and the ability to compare features of SSS 
programs to determine whether some features arc more effective than others. 

Similar to chapter 3, chapter 4 also uses both regression analysis and HLM to examine the 
nine SSS services and two organizational characteristics. Besides the use of multiple measures of 
participation, chapter 4 also differs from chapter 3 by sometimes including separate propensity scores to 
statistically adjust for the likelihood that students will receive each of the services. The primary purpose 
of adding propensity scores is to separate students’ tendency to receive services (which may in itself be 
related to student outcomes) from students’ actual receipt of services, so that measures of the relationship 
between participation and student outcomes will not be confounded by the relationship between the 
tendency to participate and student outcomes. Again, so that the reader can compare the different 
methodological choices, the statistical results are presented both with and without the use of propensity 
scores. Finally, chapter 5 attempts to synthesize the various approaches, looking at which findings are 
consistently supported regardless of the approach, which findings depend on the particular approach being 
used, and which findings (from our perspective) best describe the association between participation in 
SSS and student outcomes. 

Description of SSS 

The purposes of SSS are: 



■ To increase college retention and graduation rates for eligible students; 

■ To increase the transfer rates of eligible students from two-year to four-year institutions; 
and 

■ To foster an institutional climate supportive of success of low-income and first- 
generation college students and individuals with disabilities. 

Two-thirds of the students served by an SSS project must be low-income (defined as at or 
below 150 percent of the poverty level) 2 and first-generation college students or students with disabilities. 
The other third must be low-income or first-generation college students. One-third of the disabled 
students also must be low income. Services may include counseling, tutoring, workshops, laboratories, 
cultural events, special services for disabled students, and instructional courses that are solely for SSS 
students. 



2 In 1999, the income level was $25,050 for a family of four in the 48 contiguous states. 



3 




SSS projects have great latitude to custom-design their services to fit the particular needs of 
the institution. They thus vary considerably in terms of which specific services are offered through SSS 
and how they are organized. There is no single “SSS experience” that describes the participation of all 
SSS students. All SSS projects provide academic advising as one of their services (although even here 
there is great variation in the amount of academic advising received), while they vary greatly with respect 
to offering other services such as tutoring, labs, workshops, special instructional courses, and services 
specifically for students with disabilities. 

SSS students are more disadvantaged than the student population as a whole, not only with 
regard to the criteria used for eligibility but also in other ways that are interrelated with their 
disadvantaged background. Following arc the characteristics of the SSS students who participated in the 
national study: 3 



■ Age. SSS students entered college later than typical students. Sixty percent of the SSS 
students were either 18 or 19 years old, compared with 90 percent of all freshmen 
nationwide. 

■ Gender. Two-thirds (67 percent) of SSS participants were female, compared with 53 
percent overall. 

■ Race and ethnicity. The three largest groups were whites (41 percent), African 
Americans (31 percent), and Hispanics (22 percent). By contrast, the respective 
percentages for freshmen overall are 80 percent, 9 percent, and 6 percent. 

■ Marital status. SSS participants were less likely than the general population of 
freshmen to be currently married (11 percent versus 27 percent), and more likely to be 
never married (80 percent versus 71 percent) or divorced or separated (9 percent versus 
2 percent). 

■ Disability. Seventeen percent of SSS participants reported having some kind of 
disability. The most common was a specific learning disability, with 6 percent of SSS 
participants reporting one (compared with 2 percent among freshmen overall). 

■ High school graduation. Ninety-one percent of SSS participants graduated from high 
school (compared with 98 percent overall), 9 percent obtained a GED, and 1 percent left 
high school without receiving a diploma. 

■ Academic major. When they were freshmen, 18 percent of SSS participants selected 
health-related fields as their major, 17 percent chose business, 11 percent chose 
education, and 11 percent chose social sciences. 

■ Residence. Sixty-four percent of SSS participants attended colleges that were within 50 
miles of home, compared with 46 percent of all freshmen. 



3 More detailed information about the characteristics of the SSS participants is provided in chapter 2 and in National Study of 
Student Support Services, Report Number 2, Profile of Freshmen Participants and Project Services: 1991-92, U.S. Department 
of Education, 1994. 
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■ Financial aid. SSS participants were much more likely to receive financial aid than 
undergraduates overall (82 percent versus 42 to 45 percent, depending on the 
dependency status of the students). 



The SSS Services 

Because SSS projects vary greatly, this study collected data on the number of services that 
each student received as a freshman in nine different areas (exhibit 1-1). The services that were most 
widely received were professional counseling (80 percent of all SSS students), peer tutoring (45 percent), 
instructional courses limited to SSS students (32 percent), and workshops (31 percent), while the 
remaining services were received by 17 percent or less of the students. 

Institutions' packages of services varied, both in terms of what services were offered and 
whether the services were offered through SSS or some other mechanism. Thus, sometimes students did 
not have the opportunity to participate in some of the services listed in exhibit 1-1. At other times, the 
services may have been available, but they were provided through a mechanism other than SSS. For 
example, in order to help demonstrate that the funds for SSS were not supplanting existing institutional 
programs, an institution may have chosen to create special new services for SSS students, while also 
referring the SSS students to additional services outside of SSS that had already been offered at the 
institution. Thus, a list of the SSS services that a student received typically is not a complete list of all of 
the supplemental services that student received at the institution. 

As a rule, SSS students are in full control of determining both the types and amounts of 
services they receive, as long as the services are offered by the institutions and the students qualify for 
those services (i.e., a student must have a disability in order to receive special services for individuals 
with disabilities). At least two exceptions were noted in the case studies, however: essentially all SSS 
students participated in academic advising at the time of their enrollment, and some institutions made a 
practice of calling in students for additional advising if their early performance at the college was poor. 

In addition, the case studies revealed that SSS projects differed in their basic organization, 
and the analysis in this report focuses on two characteristics in particular. First, home-based programs 
provided a home base on campus that served the “whole student” by providing a broader range of services 
to facilitate the students’ integration on the campus and by seeing that any needed supplemental services 
were provided. Often home-based programs made special attempts to have group activities for the SSS 
students, such as cultural events or service projects. They can be contrasted with dominant service 
programs, which primarily focused on providing a single service, and all service programs, which served 
as the only (or at least primary) provider of support services at the institution. 4 Second, SSS programs 
differed in the extent to which the SSS services were blended with other services on campus. In order to 
satisfy early federal requirements (up to 1992) for nonsupplanting and nonduplication, almost all of the 
programs had ways of maintaining their unique service and population served; still, some did this by 
coordination with other service providers and some by having a more separate service delivery model. 

SSS program regulations were later changed to allow a greater blending of services, but this distinction 
was important at least at the start and proved sometimes to be important in the statistical analysis. 



4 These categories are not necessarily mutually exclusive. For this analysis, however, we chose to describe each program using 
only the single category that best described the program, rather than assigning multiple categories to a program. Further, 
because only three programs in our sample fit the all service category, and our initial investigation suggested that home-based 
programs deserved the closest analysis, we focus on the distinction between home-based programs and all other programs. 
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Exhibit 1-1. List of SSS services and the frequency that they were received in 1991-92 



Service category and description 


Types of services 


Instructional courses 

Courses were developmental in focus 
Only includes courses that were offered exclusively to SSS 
students 

Courses may or may not have been for credit 


study skills (16 percent of SSS students) 
writing (12 percent) 

developmental mathematics (11 percent) 
reading (7 percent) 
developmental English (4 percent) 
English proficiency (1 percent) 


Professional tutoring 

Designed to provide assistance with course work 
All tutors were paid, but this group was distinguished by the 
use of graduate students or faculty and staff to perform the 
tutoring 

Includes both one-to-one and group tutoring 


English (8 percent) 
general tutoring (3 percent) 
mathematics (3 percent) 
science (1 percent) 


Peer tutoring 

Similar in design to professional tutoring but performed by 
undergraduates 


mathematics (21 percent) 
English (17 percent) 
science (9 percent) 
social sciences (4 percent) 
general tutoring (3 percent) 


Professional counseling 

Provision of advice or counseling (as distinct from content 
area instruction) 

Performed by graduate students or faculty and staff 


academic counseling/ advising (60 
percent) 

personal counseling (20 percent) 
financial aid counseling (19 percent) 
career counseling (8 percent) 


Peer counseling 

Performed by undergraduates 

Otherwise similar to professional counseling 


academic counseling/advising (10 
percent) 

personal counseling (4 percent) 
financial aid counseling (1 percent) 


Labs 

Supplemental assistance in content areas 
Provided in group settings 
Similar to group tutoring 


mathematics (7 percent) 
writing (4 percent) 
reading (3 percent) 
English (2 percent) 


Workshops 

Designed to provide skill enhancement rather than content 
knowledge (e.g., study skills and orientation) 


orientation to college (18 percent) 
study skills (8 percent) 
career guidance (3 percent) 


Cultural events 

Group trips to concerts, museums, or other events 


concerts (3 percent) 
museums (1 percent) 
lectures (1 percent) 
other events (4 percent) 


Services for the disabled 

Special services, such as note-taking, counted as SSS services 
only if the services were provided through SSS (rather than 
referrals) 


counseling (2 percent) 
note takers (1 percent) 
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Findings From Earlier Years of This Study 



This report focuses on student outcomes six years after freshmen began participating in SSS. 
As part of the same longitudinal study, earlier reports discussed the effects of SSS on student outcomes 
after one year and after three years. The outcomes that were examined earlier were largely the same 
outcomes as those discussed here (effects on grade point averages, credits earned, and retention, though 
over shorter time frames), except that the one- or three-year time periods were too short to examine the 
effects on degree completion and on transfers from two-year to four-year institutions. Following are the 
main findings presented in those reports. 

■ SSS showed a small but positive and statistically significant effect on students’ GPAs, 
number of semester credits earned, and retention. The greatest effect generally 
occurred during the first year, when the most SSS services were received, but some 
SSS services received in the first year showed persisting effects in later years, and 
some services received in later years (not necessarily through SSS) also showed 
positive and statistically significant effects. 

— Students’ college GPAs were increased by a mean of 0.15 in the first year, 
resulting in a mean GPA of 2.29 on a four-point scale (e.g., “A”=4.0 and 
“C+”=2.3). In the second year, the mean increase was 0.11 (to 2.44), and in 
the first three years combined the increase was also 0.11 (to 2.59). 

— The number of semester credits earned was increased by a mean of 1 .25 (to a 
total of 20.91 credits) in the first year, 0.79 (to 20.62) in the second year, 0.71 
(to 20.58) in the third year, and 2.25 (to 73.38) in the first three years 
combined. 

— Retention was increased at the same institution by 7 percentage points (i.e., 
from 60 percent to 67 percent) for retention to the second year, and by 9 
percentage points (i.e., from 40 percent to 49 percent) for retention to the third 
year. Retention to the third year at any higher education institution was 
increased by 3 percentage points (i.e., from 74 percent to 77 percent). 

■ The average effect was small because most students received only a modest amount of 
services. Nine percent of students had only one service contact in their freshman year. 
The mean number of hours of services received in the first year was 32, and the 
median was 14. The mean for nonfreshmen was 15 hours, and the median was 6. 

■ The size of the effect depended on the degree to which students participated in SSS, 
with greater levels of participation resulting in a greater effect. 

■ The estimated effects of SSS also varied based on which particular services each 
student received and the structure of the SSS projects. 

■ The effects of SSS generally appeared consistent across different subgroups of 
students. To the extent that some subgroups showed different effects than other 
subgroups, those differences appear attributable to differences in participation levels 
rather than to differing effects of individual services. In general, students who were 
more disadvantaged tended to participate more and thus experienced greater effects 
through SSS participation. 
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■ SSS projects appeared successful in targeting those students who were most 
disadvantaged from among the overall student population. In comparison with the 
national averages for college freshmen, SSS participants tended to be older, to be 
members of a minority group, to have had lower prior academic achievement, and to 
have dependent children. Minority SSS participants composed 54 percent of the SSS 
student population, whereas minority populations represented only 25 percent of the 
total undergraduate population. 

■ SSS students received higher levels of supplemental services than did comparison 
students, including services offered outside of SSS. This suggests that SSS increased 
the amount of services obtained by students beyond what they would have received 
otherwise. This difference in service use declined substantially after the freshman 
year. For example, 63 percent of these students received tutoring at some point during 
their first three years compared with 36 percent of comparison group members. In the 
first term, 46 percent of SSS participants reported use of tutoring compared with 20 
percent of comparisons. By the spring 1994 term, 11 percent of SSS and 8 percent of 
comparisons reported use of tutoring. There was less difference between the SSS 
participants and the comparison group in levels of counseling use. 

■ Although SSS has increased greatly in size, when adjusted for inflation, the funding 
per program and per participant was less than in 1970. SSS program funding went 
from its initial funding level of $10 million in 1970 to $143.5 million in 1995. Over 
the same time period, the number of projects funded grew from 121 to 706, and the 
total number of students served by the SSS program from 30,000 to 165,561. The 
number of students served peaked at 181,368 in 1981. In constant 1990 dollars, the 
average grant size declined from $278,393 in 1970 to $174,365 in 1995. The level of 
per-participant funding (in constant 1990 dollars) was highest in 1970 at $1,123, 
declined to $507 in 1981, and reached $744 in 1995. In 1995 current dollars, funding 
per participant was $867. 

■ SSS programs are concentrated at particular types of institutions. In 1994, 
approximately 24 percent of all higher education institutions serving freshmen had 
SSS projects. Because SSS projects tended to be located in larger schools, about 34 
percent of all freshmen attended institutions having SSS projects. SSS projects tended 
to be concentrated in four-year institutions, public institutions, institutions enrolling 
more than 20,000 students, and institutions with 50 percent or more minority 
enrollment. Over 40 percent of doctoral institutions compared with 15 percent of 
baccalaureate institutions and 22 percent of two-year institutions had SSS programs. 
Relatively few highly selective institutions (19 percent) had SSS projects. 
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2. Study Design 



This chapter discusses the key concepts in the design of this study: the outcome measures 
used, an overview of the study design, the longitudinal study, and the interpretation of the data. 
Additional information about the research methodology is provided in Appendix B. 

Outcome Measures Used 



Five types of measures were used as outcome measures for the study (see Exhibit 2-1). The 
following discussion provides additional information about each variable as it applies to the SSS students. 



Exhibit 2-1. Outcome measures for the sixth-year evaluation after freshman entry in 1991-92 



Variable 


Description 


Source 


GPA (cumulative) 


Four-point scale (A=4.0) 


Postsecondary transcripts 


Total credits earned (cumulative) 


Total number of semester credits 


Postsecondary transcripts 


Retention or degree completion 


Student either earned a degree (bachelor's degree or 
higher) within six years, or was still in college in the 
sixth year 


Postsecondary 
transcripts, 1996-97 
survey of students 


At the same institution 


At any institution 


Degree completion 


Student earned the specified degree within six years 


Postsecondary 
transcripts, 1996-97 
survey of students 


Bachelor’s degree or higher 


Associate’s degree or higher 


Transfers from two-year to four-year 
institutions 


Student who originally enrolled at a two-year 
institution later transferred (within six years) to a four- 
year institution 


Postsecondary 
transcripts, 1996-97 
survey of students 



Grade point average (GPA). After six years, SSS participants in the study had a mean 
GPA of 2.3 (i.e., about a C+ on a 4.0 scale), with most students (81 percent) getting Bs or Cs as their 
average grades (exhibit 2-2). 



Exhibit 2-2. Percentage distribution of all SSS participants, by mean GPA levels six years 
after freshman entry in 1991-92 




SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Number of credits earned. On average, the SSS participants in the study earned 75.5 
semester credits over their first six years in college, or roughly the amount that would be earned through 
2.5 years of full-time enrollment. 5 About one-fourth (28 percent) had earned more than enough to 
graduate (exhibit 2-3), although they had not necessarily met other graduation requirements (83 percent of 
them had completed a baccalaureate degree or higher). The total number of credits earned depends on 
several factors, including whether or not the students had graduated front college (having graduated 
means that students must have taken a certain minimum number of credits in order to qualify, but 
graduation also is a reason for students to consider their education completed and to stop earning further 
credits), whether the student was attending part-time or full-time, and whether the student had stopped 
attending (or dropped out) for one or more years. Disadvantaged students are more likely to be part-time 
and to interrupt their education than other students, so in general they would be expected to earn fewer 
credits over the same amount of time. Measures of the amount of credits earned can vary front one 
institution to another because institutions sometimes vary in which courses are for credit and which earn 
no credits; however, this distinction is probably less important in later years than in the first years, 
because some of the introductory courses show the most variation in how they are treated. 



Exhibit 2-3. Percentage of SSS participants who earned various amounts of credits over six 
years after freshman entry in 1991-92 



More than 
1 20 credits 
28% 



90.1-120 
credrts 
11% 




60.1-90 

credrts 

16 % 



Up to 30 
credits 
27% 



30.1-60 

credrts 

18% 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 



Retention, degree attainment, and transfers to four-year institutions. Two of the 

primary goals of SSS are to help disadvantaged students remain in college and ultimately to earn degrees 
(especially four-year, or baccalaureate, degrees). These goals are interrelated since students must remain 
in college for some minimum amount of time in order to earn degrees, and once students do earn degrees, 
they are likely to stop being enrolled in college (unless they seek more advanced degrees). A third goal 
also is closely related to these goals: encouraging students at two-year institutions to transfer to four-year 
institutions so they can earn four-year degrees. Transferring to four-year institutions is related to both 
retention (because one must remain in education a longer time) and degree attainment (especially because 



5 Credits earned under the quarter system were converted to semester credits to provide comparable measures across students and 
schools. 
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degree attainment is here usually defined in terms of receiving a bachelor’s degree or higher). Because of 
the interrelationships among these three outcomes, all three are discussed together. 



Now, with six years of data available, the analysis of retention can be expanded to cover 
more years, and it is possible to examine degree attainment and transfers to four-year institutions. Even 
six years is not enough time to come to a complete conclusion for all students, however. While four years 
is considered the traditional amount of time to complete a bachelor’s degree, only about a third (36 
percent) of college graduates complete a degree in that time after starting college. 6 By contrast, about 
one-fourth (26 percent) takes more than six years to complete their degrees. Further, disadvantaged and 
minority students tend to take longer to complete their degrees than other students. While 25 percent of 
white college graduates took more than six years to complete a bachelor’s degree, 32 percent of blacks, 

35 percent of Hispanics, and 43 percent of American Indians and Alaskan Natives took more than six 
years. 7 



After six years, 14 percent of the SSS participants had completed a baccalaureate degree and 
were continuing in postsecondary education, 23 percent had completed a baccalaureate degree and were 
no longer in college, 5 percent had completed an associate’s degree and were continuing in college, 7 
percent had completed an associate's degree and were no longer in college, 20 percent were still in 
college without completing a degree, and 31 percent neither were in college nor had completed a degree 
(exhibit 2-4). This rate of baccalaureate degree completion (37 percent after six years) was higher than 
that found for the most disadvantaged students in the Beginning Postsecondary Students (BPS) study, 
which showed completion rates of 25.7 percent for those with family incomes lower than $25,000, and 
32.7 percent for those with incomes between $25,000 to $44,999. 8 



6 Thomas M. Smith, The Condition of Education 1996, U.S. Department of Education, National Center for Education Statistics, 
NCES 96-304 (Washington, D.C.: 1996), 60. 

7 Ibid., p. 215. Asians and Pacific Islanders were an exception among minorities, with only 17 percent taking more than six years 
to complete a bachelor’s degree. The three racial or ethnic categories used here are highly correlated with being disadvantaged 
and are used because statistics on the education status of these groups are more readily available. 

s National Center for Education Statistics, Descriptive Summary of 1995-96 Beginning Postsecondary Students: Six Years Later , 
U.S. Department of Education, (Washington, D.C.: 2002), 57. 
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Exhibit 2-4. Retention and attainment of degrees by SSS participants six years after 
freshman entry in 1991-92 




SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 



Degree completion alone. While retention and degree completion can be examined through 
combined measures, encouraging degree completion by itself is one of the primary goals of SSS. This 
study therefore examined participants’ degree completion rates, regardless of whether they were still in 
college. Because this analysis is based on six years of data, and disadvantaged students often take a 
longer period of time to complete their degrees, these data reflect the effects of SSS after six years, but 
not necessarily the final SSS effect. The final effect might be larger (if additional SSS participants 
complete their degrees later, due in part to their participation) or smaller (if, say, SSS provides a 
temporary advantage, but nonparticipants eventually catch up). 

This analysis uses three basic measures of degree attainment — whether participants earned a 
bachelor’s degree or higher (a specific goal of SSS), whether participants earned any degree (including an 
associate’s degree), and whether participants at two-year institutions earned any degree (again, including 
an associate’s degree). The third measure is used to better understand the effects of SSS at two-year 
institutions, since students at such schools may be less likely to earn four-year degrees. While it would be 
possible to examine attainment of associate’s degrees at two-year institutions (rather than an associate’s 
degree or higher), such an analysis might produce misleading results: students who did not earn 
associate’s degrees but did earn bachelor’s degrees would be considered “failures” in such a measure (i.e., 
they had not earned associate’s degrees), and the group of “failures” thus could include both students on 
whom SSS had no effect and students on whom SSS had a great effect. 
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Such a measure could easily lead to improper estimates of the effects of SSS. In fact, 65 
percent of community college beginners who transferred to a four-year institution did so without first 
earning a degree at the two-year institution. 9 By instead counting the attainment of any degree (i.e., an 
associate's degree or higher), all students whose degree attainment may have been affected by SSS are 
included. However, data based on two-year institutions only should be considered as less reliable, 
particularly for testing statistical significance. With fewer cases available for analysis, combined with the 
fact that some SSS services were received by only a small number of SSS students, there is an increased 
prospect that some measures of SSS may not be statistically significant based on the number of cases 
alone. In fact, the standard demographic and attitudinal variables that were incorporated in the 
multivariate analysis also performed poorly and were often statistically insignificant. Further, diagnostic 
tests indicate that the logistic regression model may be providing questionable statistics, which again may 
be due to the reduced number of cases available for analysis. 10 

Transfers from two-year to four-year institutions. Because a purpose of SSS is to 

encourage disadvantaged students to earn four-year degrees, encouraging transfers from two-year to four- 
year institutions is a subsidiary goal. Among the SSS participants in the sample at two-year institutions, 
19 percent transferred to four-year institutions. This percentage appears to be somewhat higher than 
would be expected based on other databases. According to data from the Beginning Postsecondary 
Students Longitudinal Study, 13 percent of full-time low-income freshmen at two-year institutions had 
transferred to four-year institutions within five years of starting their college careers. 1 1 

Two factors complicate the measurement of the SSS effect on transfers in this study. First, 
because the sample was not limited to two-year colleges, fewer cases were available for the analysis of 
transfers than for the preceding analyses, making statistically significant findings less likely to appear. 
Second, transfers to four-year institutions typically occur at the end of the second year (i.e., the typical 
end of a two-year program) or later, since many SSS students switched to part-time status and appeared to 
take longer to complete their academic careers. In that context, the focus on first-year SSS services may 
not necessarily be the best basis for examining transfers; it may be that decisions on transfers arc made at 
a much later time, so that first-year services are not particularly relevant. For example, SSS advisors may 
primarily concentrate on survival and general academic strategies in the first year, and shift their focus to 
assisting with transfers in later years. 

Employment. Another way of evaluating the effects of SSS participation is to examine the 
patterns of participants’ employment. If participation in SSS helps students to be more successful in 
college (as indicated by the last two chapters in this report), and if attending college helps students to be 
more successful in employment, there logically should be some connection between SSS participation 
and students’ success in their employment. 

However, several factors complicate the search for such connections. First, many students 
might be expected to take more than six years to complete their college education, so students may not yet 



9 A. McCormick and C.D. Carroll, Transfer Behavior Among Beginning Postsecondary Students: 1989-94, U.S. Department of 
Education, National Center for Education Statistics, NCES 97-266 (Washington. D.C.: 1997), vii. 

10 The models converge properly when the full database is used, but they do not consistently do so when used only for two-year 
institutions. 

11 1989-90 Beginning Postsecondary Students Longitudinal Study, Second Follow-up (BPS:90/94), National Center for 
Education Statistics, U.S. Department of Education. Low-income freshmen here are defined as freshmen whose families were 
in the bottom quartile in their socioeconomic status. 
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have had time for their employment to be greatly affected by their college attendance. Second, in some 
ways employment and education reflect two competing ways for people to spend their time, and their 
choices depend in part on their families’ financial resources. Employment may be an indicator of 
financial need rather than of success in education. Because SSS targets disadvantaged students, a high 
employment rate is likely as a response to the students’ generally greater financial needs and may 
possibly interfere with academic progress rather than being a result of it. Third, the six-year period of this 
study is too short for many students to have become established in their careers (especially considering 
that much of that time may have been consumed by education), so that students’ current careers may be 
poor indicators of their ultimate careers. Fourth, in the short term employment success sometimes is 
correlated more with seniority than education; people who have been employed continuously for six years 
may have greater temporary employment success (e.g., higher salaries due to experience and seniority) 
than people just entering employment with a college degree, even though the college degree ultimately 
may be correlated with greater lifetime earnings. Thus, if SSS encourages students to stay in college, in 
the short term the employment effects might appear to be negative. Fifth, there is no single perfect 
indicator of employment success; for example, while annual salaries are one potential measure, two 
people may be equally successful in their chosen careers and yet have very different incomes. 

Given all of these limitations, this report does not look at employment outcomes for potential 
effects from SSS. 



Overview of the Study Design 



SSS provides a relatively moderate-sized intervention in students' educational experiences, 
and the nature of the SSS experience varies from one institution to another and from one student to 
another. These qualities make it a difficult program to evaluate. One would not expect to find large 
effects on students, given the moderate nature of the SSS experience, and may not necessarily find 
consistent effects, given the wide variation in projects and levels of participation. 

Because of the potential difficulties in measuring the effects of SSS, the evaluation study 
was designed to provide multiple perspectives on SSS: 

■ A nationally representative, stratified random sample of 200 SSS projects was 
surveyed in 1991-92. 

■ Site visits were made to 50 higher education institutions, with 30 having SSS projects 
and 20 not having SSS grants. The 30 were randomly subsampled from the sample of 
200 institutions for the project survey. The 20 non-SSS sites were selected to match 
the SSS sites, based on enrollment size, geographic region, selectivity, percentage 
receiving Pell Grants, institution type (two-year, four-year), and institutional control 
(public, private). Three of the institutions later dropped out of the study, leaving 47 
institutions that became the primary focus of the analysis in this report. 

■ Project-reported data were collected from the annual performance reports submitted 
by each project, providing information on the students served and the types of services 
provided. 
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■ Other basic data were collected from the Department of Education, the Council for 
Opportunity in Education (COE; formerly called the National Council of Educational 
Opportunity Associations, or NCEOA), the Integrated Postsecondary Education Data 
System (IPEDS), and a nationally representative Higher Education Survey (HES) on 
institutions’ retention practices. 

■ A longitudinal study was conducted to compare the outcomes of 2,900 SSS 
participants and 2,900 comparison students who did not participate in SSS but had 
similar background characteristics. The students were selected from 28 of the 30 
sampled SSS projects, and 19 of the 20 non-SSS institutions. 

The Longitudinal Study 



The longitudinal study is the primary focus of this report, though data from the other 
components of the study were used to help develop the longitudinal study and are also used here to 
supplement the data and help in interpreting the statistical results. Just as the evaluation study as a whole 
sought multiple perspectives, the longitudinal study also collected data from multiple sources. It should 
be noted that this study utilizes data from the 1990s and may not capture the full nature of the SSS 
program as it operates today. 

■ Student information files provided by the 47 participating institutions provided basic 
information on student characteristics, though the amount of information that was 
available varied from one site to another. 

■ Participant service records provided detailed information on each service contact 
through SSS with the sampled students, including the types of service, length in 
minutes, date, and number of students in the service. 

■ Student surveys were conducted at three different time points: (1) to provide baseline 
measures during students' initial enrollment as freshmen at the participating 
institutions in 1991-92, (2) in the third year (1994-95) to provide updated information 
on students’ college experiences, plans, and attitudes, and (3) in the sixth year (1997— 
98) to provide further updates on the students, including potentially their graduation 
and attendance at postsecondary institutions. 

■ Student transcripts were collected at the end of the first year, at the end of the third 
year, and at the end of the sixth year to provide detailed information on students’ 
academic progress. Because many students transferred to other institutions during the 
six-year course of the study, the total number of institutions involved in the transcript 
collection increased from an initial base of 47 institutions to roughly 850 institutions 
by the end of the third year, and to roughly 1,050 by the end of the sixth year. 

The design of the longitudinal study raised a number of complex methodological issues. In 
the remainder of this section, we discuss these issues, the study design that was chosen, and the 
implications of the methodological issues for this study. 

This study used a quasi-experimental design, in which two groups of students were chosen to 
have roughly similar characteristics, with one group receiving SSS services and the other not receiving 
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such services (at least during their freshman year). By comparing the outcomes of both groups, and also 
allowing for other supplemental services that might have been received outside of SSS, the study 
developed estimates of the impact of SSS and of supplemental services in general. If exposures to SSS 
had been randomized, the study could be classified as an experimental design, but there was no 
randomization. Additional discussion of the differences between experimental and quasi-experimental 
design, and on the reasons for choosing a quasi-experimental design for this study, is provided in 
Appendix B . 



The presence of self-selection of services in SSS raised difficult but unavoidable issues for 
the evaluation study. Therefore, this study used a variety of data and statistical approaches in order to 
estimate the importance of the self-selection on student outcomes. It presents separate analyses that either 
(1) treat all SSS students the same regardless of what services they have received or (2) specifically 
model the amount and type of services that each student received. It also used a large amount of 
supplemental data on student characteristics both to statistically adjust for the interrelationship between 
those characteristics and student outcomes and to model students’ likelihood of participation. 

To monitor students’ use of supplemental services, the SSS study sites maintained detailed 
records of each service contact with the sampled students during the first year. (Students’ level of 
participation in SSS is typically highest during the freshman year, though they can and sometimes do 
continue to receive services in later years. No official service records were maintained after the freshman 
year in order to reduce the burden on the participating institutions.) Also, the student surveys in the first, 
third, and sixth year of the study were used to collect students’ self-reports on their receipt of 
supplemental services to capture information on SSS students’ receipt of non-SSS services and services 
received after the first year, and on comparison students’ receipt of any supplemental services. 

Inclusion of institutions without SSS programs. To further strengthen the study design, 
and at the request of the U.S. Office of Management and Budget, this study included 19 institutions that 
did not have SSS programs as a source for additional comparison students. This design aspect has two 
main advantages: 



■ As noted, in order for random selection to be maximally useful, the randomization 
should be applied to all relevant characteristics. It is possible that institutions that 
have SSS programs differ systematically from other institutions; for example, they 
may have a greater orientation to accepting minority or disadvantaged students, they 
may be especially well administered (e.g., in order to win an SSS grant against 
competing institutions), or they may have a strong orientation to providing students 
with supplemental assistance and working to see that all students are integrated into 
the campus community. All of these factors could potentially be related to student 
outcomes, and their importance may affect the viability of expanding SSS to 
additional institutions, as has been the trend. Some of these characteristics also might 
lead to (or arise from) SSS institutions having a fundamentally different mixture of 
students than other institutions (at least with regard to certain characteristics). By 
including non-SSS institutions in the study, one has the capacity to check for 
differences in student characteristics, and the study can better be generalized to a 
larger set of institutions. 

■ There is the potential that SSS programs may “skim” the most disadvantaged students 
in an institution, so that there is not a truly comparable set of nonparticipating students 
facing the same disadvantages within that institution. By including institutions that 
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are similar to SSS institutions except that they lack SSS programs, one has a better 
potential for finding comparison students who collectively are similar to the SSS 
participants, even if SSS students and non-SSS students are not necessarily 
comparable within any particular institution. 



Selection of comparison group. Because randomization was not used to produce 
equivalent sets of treatment and comparison students, it was necessary in some other way to select an 
appropriate group of comparison students. The comparison group was chosen by using regression 
analysis to calculate propensity scores of students' likelihood of participation in SSS based on 
demographic data that were available from the colleges. 12 The derived formulas were then used to choose 
a comparison group at each institution of students whose propensity scores showed similar distributions. 

As shown in Exhibit 2-5, the propensity scoring was useful in selecting students who were 
more like the SSS participants than like typical students, but the differences were only partially overcome. 
In particular, SSS students tended to be more disadvantaged than the students in the comparison group; 
for example, they were more likely to have household incomes of $10,000 or less (30 percent versus 15 to 
17 percent) and to have parents with less than a high school education (e.g., 35 percent versus 17 to 20 
percent with regard to the father's education background). 13 The primary reason the SSS students and the 
comparison group were not more similar appears to be the lack of adequate data for fully comparing the 
two groups of students. For example, some institutions did not have information about the race or 
ethnicity of their students or about the students’ finances, especially for those who were not receiving 
assistance. Thus, the fewer the items that were available, the less powerful were the propensity models, 
and most institutions were able to supply only a limited amount of data. Also, as revealed by case 
studies, it was sometimes difficult to select comparable non-SSS students because that the SSS programs 
were so highly targeted at a few institutions that there were no comparable non-SSS students with similar 
characteristics. The number of SSS students at such institutions, however, was sufficiently small as to 
have only a minor effect on the overall averages. Also, the students in the highly targeted programs were 
actually less likely to be disadvantaged than the other SSS students (e.g., 77 percent were white, 
compared with 35 percent of other SSS students), so that targeting was not the major source of the 
differences between the SSS and comparison groups. Whatever the cause, there were systematic 
differences between the SSS participants and the comparison group, with the SSS participants being more 
disadvantaged. The analysis therefore required the use of statistical adjustments to correct for these 
differences. 



12 The formulas that were used for calculating propensity scores varied from one institution to another because institutions 
differed in the amount of information that they were able to provide for use in the statistical model. 

13 The cited differences are all statistically significant at the 0.01 level based on t-tests. 
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Exhibit 2-5 

Comparison of SSS students with comparison group and all freshmen, by student characteristics 



Student characteristic 


SSS 


Comparison 


group 


All 

freshmen 


SSS schools | 


Non-SSS 


Demographic data 










Percent male 


33% 


40% 


40% 


47% 


Never married 


81% 


87% 


86% 


71% 


Have dependent children 


22% 


15% 


15% 


24% 


Percent white 


42% 


50% 


55% 


80% 


Over 100 miles from home 


18% 


19% 


24% 


36% 


Age 1 9 or under 


75% 


79% 


70% 


92% 


Household income $10,000 or less 


30% 


17% 


15% 


7% 


Other language besides English spoken at home 


31% 


23% 


17% 


- 


Understand English very well 


71% 


88% 


83% 


- 


Speak English very well 


59% 


82% 


69% 


- 


Write English very well 


48% 


78% 


67% 


- 


Read English very well 


60% 


82% 


72% 


- 


Parental information 










Lived with father/male guardian 


72% 


77% 


75% 


— 


Lived with mother/female guardian 


94% 


96% 


97% 


— 


Father was manager/proprietor 


18% 


24% 


24% 


- 


Mother in service occupation 


51% 


39% 


36% 


- 


Father had less than high school education 


35% 


20% 


17% 


12% 


Mother had less than high school education 


30% 


19% 


15% 


9% 


Academic information 










Have taken SAT or ACT 


81% 


87% 


85% 


— 


SAT Verbal (mean) 


399.0 


455.3 


445.0 


422 


SAT Math (mean) 


428.7 


463.0 


462.4 


474 


ACT Composite (mean) 


18.8 


20.2 


20.4 


21 


"A" as average grade 


10% 


11% 


17% 


24% 


Years of math (mean) 


3.3 


3.4 


3.4 


- 


Took courses at other college 


29% 


36% 


40% 


— 


Credits earned at this institution (mean) 


21.1 


21.5 


22.3 


— 


Full-time student 


90% 


91% 


90% 


— 


Work-study job 


16% 


13% 


11% 


— 


College finances 










Received financial aid 


82% 


73% 


73% 


45% 


High school counselor helped assemble 


26% 


19% 


19% 


— 


College counselor helped assemble 


27% 


26% 


16% 


- 


Self-assembled 


50% 


68% 


60% 


— 


Listed as dependent by parents 


43% 


62% 


64% 


- 


Received assistance of $600 or more 


27% 


43% 


48% 


— 


Financing college is major concern 


41% 


34% 


31% 


- 


Attitudes 










BA/higher as highest planned degree (this college) 


67% 


72% 


68% 


67% 


BA/higher highest planned at any college 


91% 


95% 


95% 


90% 


Definitely able to complete college 


71% 


79% 


80% 




Academic ability above average 


43% 


59% 


57% 


54% 


Drive to achieve above average 


65% 


72% 


73% 


67% 


Emotional health above average 


58% 


67% 


71% 


55% 


Leadership ability above average 


51% 


62% 


65% 


50% 


Mathematical ability above average 


30% 


42% 


41% 


37% 


Physical health above average 


57% 


74% 


73% 


56% 


Popularity above average 


40% 


52% 


57% 


38% 


Intellectual self-confidence above average 


57% 


71% 


71% 


51% 


Social self-confidence above average 


56% 


66% 


66% 


45% 


Writing ability above average 


38% 


56% 


54% 


40% 


—Data not available. 



SOURCE: U.S. Department of Education, Planning and Evaluation Service, National Study of Student Support Services (SSS), Service Record 
Analysis, 1991-92. 
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It is important to note that though the distribution of characteristics among the SSS and 
comparison groups was different, both groups were diverse (though disadvantaged) and there was no 
discernable group of SSS students who were so disadvantaged that the comparison group lacked such 
students. If such a subgroup did exist, then statistical models might be limited in their ability to discuss 
program effects for such students. Fortunately, the two groups did overlap in their coverage of students, 
thus improving the chances that the results would be generalizable and that multivariate regression 
models could successfully adjust for these differences. The pooling of data across multiple institutions is 
especially helpful in this regard because it helps to provide groups of students with similar characteristics 
in both the SSS and comparison groups even if SSS students were highly distinct in a few institutions. 

The inclusion of students from institutions without SSS programs also helped by providing institutions in 
which skimming of the most disadvantaged students by SSS was not an issue. In order to maximize 
diversity among both SSS students and comparison students, the analyses were conducted with all of the 
institutions combined. A limited number of analyses were conducted on an institution-by-institution basis 
in the analysis after three years, but these models were considered inferior both because of the less 
adequate coverage of student characteristics and the reduced number of cases available for analysis. The 
multivariate regressions did include variables to measure some institution characteristics (whether the 
institutions were two-year, four-year, or doctorate-granting institutions, whether the SSS program 
included home -based or blended services, and whether the institutions had SSS programs), and the use 
HLM did model differences in variances on an institution-by-institution basis). 

Exhibit 2-5 also shows that there generally were only small differences between the 
comparison students based on whether they were at SSS or non-SSS institutions. Because statistical 
adjustments for the differences between SSS and non-SSS students are required in any case, and these 
adjustments can also compensate for these small differences between the two types of institutions, the 
distinction between the two comparison groups is unnecessary. Accordingly, this report combines both 
groups of comparison students into a single group. 

Intent to treat. One of the critical issues in conducting an evaluation can be the 
determination of which people are considered to have received treatment. For example, in a medical 
study, some patients might drop out because of the side effects they experience from the medication being 
studied. If those patients are ignored, then one would not have a proper measure of either the frequency 
of side effects or of the success of the medication. The solution is to consider the initial enrollment of a 
person into a treatment group as being the critical factor (i.e., that there was an “intent to treat”) even if a 
person does not receive the full treatment. However, sometimes exceptions are appropriate, and it can be 
difficult to know precisely where to draw the line. Using another medical example, many antibiotics 
require multiple doses (e.g., often over a 10-day period), and not only is it ineffective to terminate the 
medicine prematurely, but it may also be counterproductive (if the infection is not eliminated completed, 
the bacteria may develop a resistance to the antibiotic and be harder to treat afterward). Should a patient 
who received only a partial treatment still be considered as part of the treatment group? The answer, 
which depends on whether the withdrawal was in some way related to the treatment or disease, may not 
always be straightforward. 

Because SSS allows students to set their own levels of participation, and because the SSS 
programs also vary from one institution to another, the actual amount and types of services received vary 
greatly. Many students might be considered as dropouts from the program, whether based on the number 
of contacts or the hours of contacts: 9 percent received only a single service contact over the entire 
freshman year, 7 percent received less than 1 hour of total services in the first year, and 22 percent 
received between 1 and 5 hours of services. Though it is conceivable that even a single contact might be 
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sufficient to resolve a key intellectual problem or impart a needed skill, it would not be surprising if most 
or all of these students experienced little effect from their participation in SSS. In fact, many education 
policies are based on the assumption that the amount of time devoted to education is important, including 
many reform efforts aimed at increasing the time devoted to reading and mathematics, and even the way 
that graduation requirements are specified (in terms of the number of hours or credits of instruction 
received). Particularly if the students dropped out because they perceived the program as providing no 
benefit, it is reasonable to consider the students as “failures” in the sense that SSS failed to impart needed 
skills or to motivate students sufficiently to benefit from the SSS offerings. On the other hand, these 
students also consumed few of the resources in terms of the dollars spent for supplemental services. If 
one were conducting a cost-benefit analysis, one might even argue that allowing dropouts actually 
increases the program's efficiency by reducing the use of resources for those who do not benefit from 
them (e.g., assuming either that a reason that students drop out is that they do not perceive a program as 
being effective, or that the factor that leads to nonparticipation — such as low motivation — would lessen 
the likelihood of the student being helped). Thus, though it is important if some students are not helped 
through SSS, it would not necessarily be a complete failure if some students are helped and some are not, 
particularly if the resources are concentrated on those who are helped. 

The question of how to measure participation is important because of how statistical 
significance is calculated. If some students benefit from SSS and other students do not, then the average 
benefit across all students may or may not be statistically significant when a dichotomous measure of 
participation is used, depending on the size of the benefit, the distribution of any benefits across the SSS 
population, and the number of students in the study. Estimates of the average benefit also would not 
provide any information on the differential effects of SSS. On the other hand, if there is an a priori means 
of predicting which students would benefit, and separate tests are run for each group, then the added 
precision is more likely to result in statistically significant findings for those students who benefit . 14 Two 
such methods are examined in this report: differentiating among students based on the amount of services 
they received, and also on the specific types of services they received. Potentially these two measurement 
tools can provide more precise information about which students and how students are helped. 

In order to address these conflicting needs to consider dropouts as potential failures of the 
program but also to test for differential effects among students, this study uses multiple statistical 
measures of participation to provide multiple perspectives. More specifically, 

■ The study uses both dichotomous (yes or no) measures of participation that group all 
SSS participants together and continuous measures that differentiate among students 
in terms of the amount and types of services they received. 

■ Regardless of the measure used, average estimates of the total effect always include 
all SSS participants, including those who might be considered as dropouts. 



14 In statistical analyses, it is possible (but incorrect) to first identify those students with superior outcomes, and then calculate 
“impacts” just for these students. This approach could result in making even random differences in outcomes appear to support 
findings of positive and statistically significant outcomes. Thus, it is critical not to separate students based on their outcomes 
but rather to use prespecified and independent measures of participation. Differentiating among students based on the amounts 
and types of services they have received is one such a priori approach and has been part of this study since the initial design 
phase, when the decision was made to collect such information on the students. 
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■ Some measures in this report provide estimates for those students who a priori might 
be considered to be most likely to have improved outcomes based on their 
participation in SSS. 

However, the use of continuous measures of participation based on the amount of services 
received does create a risk of conflating participation with motivation or need (or other factors that might 
influence participation). This topic is discussed in greater detail in the section that immediately follows. 

Separating motivation and need from participation. To the extent that students’ levels of 
participation are based on student characteristics that themselves may be correlated with student 
outcomes, there is a risk that the measure of participation may serve as a surrogate for these other factors, 
and that any statistical association that is found between participation and student outcomes actually is 
due to these other factors and not participation in SSS itself. For example, students may participate more 
if they are highly motivated to succeed in their education, and this high degree of motivation might be 
associated with improved student outcomes. Alternatively, students may participate more if they had a 
high degree of academic need, and that neediness might be associated with poor student outcomes. These 
factors work in opposite directions but either could lead to misleading statistical results. A high 
association between motivation and participation could result in making participation highly correlated 
with positive student outcomes, thus overstating the effects of participation on student outcomes. 
Alternatively, a high association between academic need and participation could result in making 
participation highly correlated with poor student outcomes, thus understating the effects of participation 
on student outcomes (or, at the extreme, resulting in a negative association between participation and 
outcomes, so that it would appear that participation is associated with reduced student outcomes). 

To investigate this issue, we attempted to predict students’ levels of participation using a 
variety of statistical models: 

■ Logistic regressions to predict whether or not a student is a participant in SSS; 

■ Multiple regression to predict the total number of hours of services received; and 

■ Multiple regressions to predict the number of hours of services received for each 
specific service. 

We found that participation is a complex phenomenon that is related both to measures associated with 
academic need and measures associated with motivation. We also found that there is no single propensity 
to participate that works uniformly across all types of services, but rather that participation in one type of 
service often shows little or no relationship to participation in another (exhibit 2-6). 
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Exhibit 2-6. Correlation of the number of hours received of each SSS service as freshmen in 1991- 
92 with the number of hours received each other SSS service in the same year 



Service 


Instruc- 

tional 

courses 


Profes- 

sional 

tutoring 


Peer 

tutoring 


Profes- 

sional 

counseling 


Peer 

counsel- 

ing 


Work- 

shops 


Labs 


Cultural 

events 


Services 

for 

disabled 


Instructional 

courses 




0.08 


-0.01 


0.28 


-0.12 


-0.07 


0.23 


0.03 


-0.04 


Professional 

tutoring 


0.08 




0.40 


0.22 


0.29 


0.13 


0.24 


0.09 


0.19 


Peer 

tutoring 


-0.01 


0.40 




0.22 


0.36 


0.05 


0.13 


0.12 


0.10 


Professional 

counseling 


0.28 


0.22 


0.22 




0.01 


0.15 


0.57 


0.32 


0.02 


Peer 

counseling 


-0.12 


0.29 


0.36 


0.01 




-0.02 


0.06 


0.08 


0.01 


Workshops 


-0.07 


0.13 


0.05 


0.15 


-0.02 


-- 


0.23 


0.33 


0.10 


Labs 


0.23 


0.24 


0.13 


0.57 


0.06 


0.23 


— 


0.23 


0.04 


Cultural 

events 


0.03 


0.09 


0.12 


0.32 


0.08 


0.33 


0.23 




0.23 


Services for 
disabled 


-0.04 


0.19 


0.10 


0.02 


0.01 


0.10 


0.04 


0.23 





SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 



Based on these findings, students’ propensity to participate is not a monolithic force, and to 
the extent that measures of participate act as surrogates for these other factors, it is not clear whether there 
would be a bias or the direction of that bias. 

To lessen the potential for bias, this study used several procedures: 

■ One of the measures of participation is dichotomous, so that SSS participants with 
high propensities to participate will not be treated separately from SSS participants 
with low propensities. (However, even a dichotomous measure can be subject to bias 
if there are different propensities between SSS and non-SSS students.) 

■ Multivariate models were used to adjust for students’ demographic differences, 
academic background, and attitudes. 

■ Separate propensity measures were created and included in some multivariate models 
to further adjust statistically for differences in the propensity to participate. 

Services received outside of SSS. One of the complicating factors for this study is that 
institutions also offer supplemental services through other means than SSS. In fact, one SSS program 
may offer a specific service through SSS, while another institution refers students to an equivalent 
program outside of SSS. Further, non-SSS students often receive services that are equivalent to those 
provided in SSS. In the longitudinal sample for this study, 91 percent of the SSS students and 53 percent 
of the comparison students received services outside of SSS. 

One potential approach to the outside services is to ignore them, based on the logic that the 
purpose of the evaluation is to determine what extra difference is made by SSS. However, then the study 
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would not be measuring the effects of SSS (versus not receiving services) but rather the comparative 
effect of SSS relative to other supplemental services, which would be risky. For example, this approach 
in a medical setting could lead to saying that one antibiotic is not effective because its cure rate for strep 
throat is little different from that of another while, in fact, both may be effective. Thus, failing to account 
for alternative treatments could lead to incorrect inferences. Presumably, policymakers do not expect 
that, say, peer tutoring offered through SSS must be superior to peer tutoring offered through other 
sources but, rather, are satisfied if peer tutoring (or any other SSS service) offered through SSS is 
effective. Thus, to accurately assess the association between SSS participation and student outcomes, it is 
necessary to adjust for the receipt of other services. 

All of the models in this analysis therefore adjust in some way for students’ receipt of 
supplemental services outside of SSS. The method that is used depends on how SSS participation is 
measured in order to provide a consistent approach. When dichotomous measures of SSS participation 
are used, a dichotomous measure of non-SSS participation is also used, and when multiple measures of 
SSS participation are used, multiple measures of non-SSS services are also used. Unfortunately, because 
essentially all of the SSS participants also received non-SSS services, the models using dichotomous 
measures are not very powerful with respect to distinguishing effects received through SSS from those 
received from supplemental services in general. There are two main alternatives, but each encounters the 
same difficulty: 



■ The measure of participation in non-SSS services could be assigned a value of 1 for 
all comparison students receiving such services, and 0 for all other comparison 
students and also for all SSS students. If supplemental services have positive effects 
on student outcomes, then both the measure of SSS services and this measure of non- 
SSS services should be positive and statistically significant. However, because the 
measure of non-SSS services would not include any non-SSS services received by 
SSS students, the measure of SSS services would include the average effects of the 
non-SSS services as well as the effects of the SSS services. The measure therefore 
would tend to overstate the effects of the SSS services. If there is a sizable difference 
between the effects for SSS students and non-SSS students receiving supplemental 
services, one might be tempted to infer that the added benefit was associated with SSS 
participation; however, the measure would not be adequate to prove this because there 
would be no way of separating the SSS and non-SSS services received by the 
participant. 15 Similarly, if the estimated effects are basically equivalent, one might be 
tempted to infer that both groups received an equivalent package of services, but the 
model would not indicate whether the SSS services were an important part of the 
package. 

■ Alternatively, the measure of non-SSS supplemental services could be assigned a 
value of 1 for both SSS and non-SSS students who received services outside of SSS, 
and 0 otherwise. This approach has the advantage that the measure of SSS services 
should in principle show only the effects of SSS and not of the other services. If a 
substantial number of SSS students did not receive non-SSS services, such a measure 
might be successful. However, because 91 percent of SSS students also received non- 
SSS services, the two measures of services are largely identical for SSS students, and 



15 One cannot assume that non-SSS services have the same effect on SSS students as on non-SSS students because the measure 
does not indicate whether both groups received equivalent amounts of non-SSS services. 
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the statistical model is unlikely to be able to differentiate between the two sources of 
services. Suppose, for example, that both SSS students and those non-SSS students 
receiving supplemental services ultimately received the same amount of services, 
though from different sources, and therefore showed the same total effect with regard 
to the supplemental services. Because the single measure of non-SSS services would 
largely be effective in identifying almost all students who received supplemental 
services, it would tend to show the full value of all supplemental services combined, 
while the SSS variable would show no added benefit of SSS services. The lack of 
differentiation between the two measures for SSS students would probably result in 
the SSS measure being statistically insignificant, even if the SSS services made a 
positive difference. 

For this report, the second alternative was chosen because it potentially provides a greater ability to 
differentiate between SSS and non-SSS services. However, it is likely to understate the effects of SSS 
because some of effect is likely to instead be captured through the other measure of supplemental 
services. 



By contrast, when multiple measures of SSS participation (and non-SSS participation) are 
used, there is a much greater ability to differentiate among students. For example, while 91 percent of 
SSS students received some non-SSS supplemental service, the percentage receiving non-SSS peer 
tutoring (or whatever service being considered) is much smaller. As a result, there is a greater ability to 
differentiate between the effects of SSS and non-SSS services. 

Services received after the first year. To reduce burden on the institutions, the SSS 
projects only collected detailed information on students’ service contacts during the first year. Thus, all 
information that is available on services received in later years is based on students’ responses to the 
questionnaires. Students were not asked about the source of those services, both to reduce the 
questionnaire length and complexity and because students were not expected to know. 16 Thus, while this 
study is able to examine whether supplemental services after the first year are associated with improved 
student outcomes, it is not able to examine the effects of SSS services in later years. 

As with both the SSS services and the first-year services received outside of SSS, the 
treatment of these services varies depending on the particular' model being used. When participation in 
SSS is treated as a dichotomous variable, then these services are also measured through a single separate 
variable for all services received after the first year. 17 As with the measure of non-SSS first year services, 
there is substantial overlap between this measure and the other two measures of supplemental services: 

80 percent of SSS students received supplemental services, along with 82 percent of students receiving 
non-SSS first year services. Thus, if either this measure of later-year services or the measure of non-SSS 
first year services is positive and statistically significant, the variable may be capturing some of the SSS 



16 One reason is that students might think only of the services as being provided by the institution, rather than by a particular 
office within the institution, and another is that SSS projects sometimes create special institution-specific names for the 
projects, so that students might not know that the projects were SSS projects even if they remembered the specific source of the 
services. 

17 Another option would have been to use a single dichotomous variable to account both for first-year non-SSS services and all 
later services. This latter option was not chosen because the other first-year services can be distinctly labeled as non-SSS 
services, while the services in later years cannot, and also because such a large proportion of students would have received 
either a first-year non-SSS service or a later year supplemental service that the variable would provide little capacity to 
distinguish one student from another. 
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effect, possibly making the measure of SSS services statistically insignificant. If so, the lack of statistical 
significance for the SSS variable would not necessarily indicate that SSS had no effect but, rather, that no 
additional benefit of SSS could be identified beyond that benefit that was already captured in the other 
two measures of supplemental services. 

Especially when the dichotomous measure of non-first-year services is used, there is some 
risk that the variable acts as a surrogate for other factors that may be correlated with academic success. 
For example, the fact that a student received services after the first year necessarily requires that the 
student “persisted” for at least one year. 18 Among the combined group of SSS and non-SSS students in 
the study, 92 percent of those who persisted also received at least one supplemental service in those later 
years, making these two measures largely similar. Alternatively, the variable might also be considered a 
measure of academic need and motivation because it applies to students who have chosen to receive 
supplemental services. Still another possibility is that to the degree the measure is related to persistence, 
it may also be modeling some of the effects of SSS participation rather than allowing the SSS measure to 
show all SSS-related effects; this might be an issue mostly when examining retention, and the estimate of 
the SSS effect might therefore be an underestimate. 

The problem is especially an issue for the dichotomous models because then the measures of 
persistence and receipt of supplemental services are the most similar. When the measure of supplemental 
services is divided into multiple continuous measures, the greater amount of variation provides a better 
capacity for relating specific services to variations in student outcomes. As an adjustment in all of the 
models (i.e., using dichotomous measures and using multiple continuous measures of participation), an 
additional dichotomous variable is added that measures whether students attended higher education at any 
time during the five years after 1991-92. Thus, the changes in outcomes that are associated with 
persistence are measured through this variable, while the measure of the receipt of supplemental services 
will only capture any additional changes in student outcomes that are associated with the receipt of 
supplemental services. 

A weakness of this approach, however, is that the model is still somewhat recursive: SSS 
participation may affect persistence, which in turn may affect the student outcome being examined, so 
that the total effect associated with participation in SSS may be some combination of the estimate for SSS 
participation plus some component of the estimate for persistence. 19 Thus, the models may tend to 
understate the effects of SSS. Conceivably, one could create a latent variable to separate that component 
of persistence that is associated with participation in SSS from the level of persistence that would appear 
in any case; however, given the wide overlap between the three measures of supplemental services (first- 
year SSS services, first-year non-SSS services, and later services from any source), the dichotomous 
models would not be well designed for measuring the effects of SSS even after creating the latent 
variable. The dichotomous models are primarily useful for testing whether supplemental services (of 
some kind) are associated with improved student outcomes, while the more precise measures of 
participation used elsewhere in this report are the best for estimating the effects specifically associated 
with participation in SSS. 



1S This is a liberal definition of persistence because it includes students who stopped out for a time and then later returned to 
postsecondary education. It is therefore not the same as retention to the second year. 

19 The problem would remain even if the measure of persistence is omitted because the measure of services received in later years 
would then have much the same effect. The only way to completely avoid this issue is to avoid any reference to services 
received in later years, but that would create a different problem: SSS participants would be compared to other students as if 
the other students had not received supplemental services, and the estimated effects of SSS (for some students) would therefore 
be the estimated difference between SSS and other services rather than the estimate difference between SSS and no services. 
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The three types of services (first-year SSS services, other first-year services, and later 
services) all show substantial overlap, especially when measured through dichotomous variables. In fact, 
only 2 percent of SSS students did not receive some type of supplemental service besides first-year SSS 
services. 



Because of the inability to distinguish between SSS and non-SSS services received after the 
first year, parts of this report focus on the more general question of whether the receipt of supplemental 
services is associated with improved academic outcomes. This general question has policy relevance for 
SSS. First, if supplemental services are found to improve student outcomes, then it is logical to assume 
that supplemental services provided through SSS also improve student outcomes. That is, there is no 
clear reason why services provided outside of SSS should be superior to SSS. If anything, given that 
institutions must win the SSS grants through a competitive process while the other services may not go 
through such a process, there are reasons to expect the SSS services to have better quality control. 

Second, to the extent that later-year services are found to be important, that information could lead SSS 
programs to increase the level of services provided in later years, thus potentially increasing the effects of 
SSS. Third, because part of the function of SSS is to refer students to needed services even if those 
services are outside of SSS, the effects of those other services is relevant to SSS and in some ways might 
be attributed to SSS. In fact, SSS students received a higher level of services than the comparison group 
both in the first year and also in next two years (Exhibit 2-7). The rates of receiving services were 
roughly comparable for both groups after 1993-94, except that the SSS students received more tutoring. 
Unfortunately, the study did not provide data on why the SSS students received more services (e.g., 
whether it was through the influence of SSS or for other reasons such as greater academic need). 

Exhibit 2-7. Percent of SSS and non-SSS students receiving services in the six-year period starting 



with freshman entry in 1991-92 





Non-SSS 


SSS 


First-year services from any source 






Services for disabled 


1 


8 


Counseling services 


50 


93 


Classroom instruction 


20 


74 


Cultural events 


16 


34 


Tutoring 


25 


73 


Services for limited-English ability 


1 


6 


Student orientation 


82 


85 


Referrals to agencies/resources 


12 


26 


Services after the first year 






Counseling in 1992-93 


41 


48 


Tutoring in 1992-93 


18 


30 


Counseling in 1993-94 


37 


40 


Tutoring in 1993-94 


12 


18 


Tutoring after 1993-94 


30 


38 


Advising/counseling after 1993-94 


51 


51 


Services for disabled after 1993-94 


2 


5 


Field trip after 1993-94 


23 


20 


Contact with support services after 1993-94 


50 


48 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Hierarchical clustering of data. Both the SSS participants and comparison group students 
were chosen by first selecting the institutions and then selecting the applicable students in the institutions. 
Though highly efficient, this selection method violates the classical statistical assumption that each 
student observation is independent of the others. To some degree, the inclusion of variables at the 
institution level within the regression models helps to adjust for such clustering. Still, the nesting of 
students within schools creates the potential that standard multivariate regression may not properly 
estimate the variances. This report therefore used both standard multivariate regression and HLM 
modeling so that readers may compare the results using both approaches. 

To develop the HLM modeling approach, we first used HLM to examine several preliminary 
multivariate models, using a variety of measures and models. HLM was not able to estimate a solution 
unless most of the variables were treated as fixed. We therefore used a stepwise procedure of adding 
variables to test for each variable whether including a residual provided significant benefits. Based on 
these tests, we determined that for all HLM models for this report, only the intercept would not be treated 
as fixed. All of the independent variables were centered around the grand mean in order to increase the 
stability of the model. The HLM software that we used does offer an option for limited dependent 
variables, but this approach generally did not converge to a solution. Thus, ordinary HLM models were 
used instead for all outcome variables, whether or not they had limited dependent variables. 

Number of cases. As noted, the longitudinal study was of 2,900 SSS participants and 2,900 
comparison students. Because of nonresponse, the actual database for examining six-year student 
outcomes consisted of 5,055 students, with 2,380 SSS participants and 2,675 comparison students. 
However, as shown in Exhibit 1-1 the number of students receiving any particular service varied widely, 
and the receipt of some services was relatively rare. The wide variations occurred for a variety of 
reasons: some services were not relevant for all students (especially services for the disabled), institutions 
varied in the packages of services they provided, and students could choose which services they received. 
Because of these valuations, and because the amounts of services received often were also small (so that 
only small effects were likely), modeling the effects of specific supplemental services sometimes was 
hampered by the lack of adequate valuation. Sometimes the finding of statistically significant results for 
one service and insignificant results for another may be based more on the numbers of students receiving 
each service rather than indicating that one service was more effective than another. The small number of 
students receiving some services could also be a reason why general measures of supplemental services 
(that do not distinguish between SSS and non-SSS students) would perform better than measures of SSS 
services, because the general measures would have roughly twice as many students receiving services. It 
especially imposed limitations on those analyses that were limited students at two-year institutions; that 
is, models to predict transfers from two-year to four-year institutions and to predict degree completion at 
two-year institutions were tested using substantially fewer students and the models performed less well. 
Many variables that were statistically significant when all students were included were not significant 
when restricted to students at two-year institutions, and the models did not always converge to a solution. 
Exhibit 2-8 shows the number of students receiving each SSS service for students in the dataset. 
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Exhibit 2-8. Number of students receiving SSS services in 1991-92 in database used for outcomes 
analysis 



Category of student 


Instruc- 

tional 

courses 


Profes- 

sional 

tutoring 


Peer 

tutoring 


Profes- 

sional 

counseling 


Peer 

counseling 


Labs 


Work- 

shops 


Cultural 

events 


Services 
for the 
disabled 


All students 


746 


384 


1,082 


1,892 


318 


390 


723 


226 


63 


At two-year 
institutions 


110 


56 


250 


415 


23 


157 


90 


8 


11 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 
Examination of Student Outcomes After Six Years, 2010. 



Presentation of effect estimates. The regression estimates are presented here in two 
different formats: as regression coefficients to aid researchers to understand the structure of the models 
and to replicate the results, and as effect estimates to aid policymakers in applying the findings, because 
the regression estimates often do not translate easily into effects. (One difficulty is that for some models, 
one must know the amount of services that students received in order to interpret the regression 
coefficients; another is that when logistic regression is used, neither the regression coefficients nor the log 
odds ratios are easy to interpret.) 

Normally, estimates of program effects would be presented by indicating both what students 
would have achieved if they did not participate in SSS and what they did achieve after participating in 
SSS. Unfortunately, because the comparison group was more advantaged than the SSS participants, and 
since students in the comparison group often also received supplemental services, there is no pure 
measure of what outcomes would be expected if students had not participated in SSS. There are actual 
measures of final student outcomes after six years, but the alternative outcomes if students had not 
participated in SSS could only be estimated through statistical models. Moreover, because this report 
uses multiple models as a tool for examining the implications of the methodological choices involved, 
there are multiple estimates of the effects of SSS from which to choose. The results across the various 
models are often highly consistent but not identical. Therefore this report presents only the estimated 
effects of the services along with the final measured outcomes. Readers can calculate what students 
would have achieved without SSS by first deciding which model appears most trustworthy, and then 
subtracting the estimated effect from the final outcomes. 

Negative and statistically significant findings. In some cases, SSS or other supplemental 
services showed negative but statistically significant findings. As a general rule these findings seem 
implausible, though there are situations in which negative findings could be the result of interactions 
between variables. 



■ For example, if SSS helps students to graduate, this could lead to their no longer being 
enrolled in higher education; thus, one might simultaneously find positive effects on 
degree attainment and negative effects on retention. To handle this particular 
situation, this report primarily focuses on a composite measure (i.e., students either 
received a degree or were retained in higher education) as the best measure in this 
area. 

■ Similarly, in the short term it is not clear how to interpret the relationship between 
SSS and employment; for example, those students who dropped out of higher 
education might be the ones who have had the most time to get established in their 
careers, possibly leading to higher employment rates and higher salaries. Thus, again 
a positive effect on retention could be correlated with an apparent negative effect on 
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employment. Because of the difficulty of interpreting employment outcomes in the 
short term, and because employment outcomes are not specifically a goal of SSS, this 
report does not look at such outcomes. 

■ More generally, negative findings can be the result of differences between the SSS 
and comparison groups. To the extent that the SSS students were more 
disadvantaged, one would expect them to have worse academic outcomes unless they 
receive supplemental services that compensate for their disadvantages. If the 
statistical models do not adequately adjust for these disadvantages and the effects of 
SSS are not sufficiently large to make up for the differences between the two groups, 
then one could find negative correlations between SSS participation and student 
outcomes. In earlier analyses, this particularly appeared to be a factor with regard to 
first-year professional counseling received by SSS students because of the way in 
which it was administered. While some models in this study do show positive 
outcomes from counseling (especially for counseling that is not specifically associated 
with SSS), first-year counseling through SSS often showed negative relationships to 
student outcomes. The reason appears to be that some SSS institutions proactively 
brought in students for counseling if they were doing badly in their academics; this 
action had the statistical effect of making high participation in counseling correlated 
with poor academic outcomes. 

All three of these explanations could result in underestimating the positive effect of SSS, either because 
of a failure to model a positive contribution from SSS (such as improved retention) or a failure to fully 
adjust for students’ disadvantages. 

Of course, one could generate alternative hypotheses as to why SSS might be harmful. For 
example, the tutors could be so poorly qualified that they teach incorrect material, or the sessions could be 
so ineffective that they take time away from more constructive uses of students’ time. These latter 
explanations seem implausible, though, both because the institutions won competitive SSS grants based 
on the quality of their proposals, and because the students would be unlikely to keep returning for 
additional sessions if they found the sessions not to be helpful (or even counterproductive). 

For completeness, and so that readers may be aware of such indicators of inadequacies in the 
models, all negative regression coefficients and low odds ratios (i.e., odds ratios below 1.0) for SSS 
variables are included in the tables if they were statistically significant. However, because they are not 
considered reliable indicators of the effects of SSS, they are not included in the effect estimates. That is, 
the statistical findings are always presented through a combination of two tables, with the first showing 
regression coefficients and odds ratios and the second showing effect estimates. The negative findings 
are included in the first, but not in the second. For the third-year report, we also examined alternative 
models in which the SSS variables with negative coefficients were dropped from the models to see if the 
estimates for the other SSS variables would be affected; both types of models produced similar results, 
suggesting that the dropping of the negative coefficients does not artificially result in high effect 
estimates. 
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3. Research Findings Based on Dichotomous 
Measures of Participation 



Highlights of Findings 



■ Exhibit 3-1 summarizes the major results from the statistics presented in this chapter. 
There were consistent and statistically significant improvements in students’ academic 
outcomes if they received supplemental services in general, whether using regression 
or HLM analysis. By contrast, the regression analysis showed positive effects from 
SSS participation only with regard to cumulative GPA, and the HLM analysis, which 
better accounts for the clustering of students in institutions, showed no effects that are 
specifically associated with SSS. However, the dichotomous models lack sufficient 
power to discriminate among the different types of supplemental services, and the 
effects that were identified for supplemental services may include effects from SSS. 

Exhibit 3-1. Summary of findings for each outcome using dichotomous measures 



of participation, by method used: Six-year outcomes after freshman 
entry in 1991-92 



Outcome measure 


Regressions 


HLM j 


First-year services 


Later 

services 


First-year services 


Later 

services 


SSS 


Other 


SSS 


Other 


GPA 


■ 




■ 






■ 


Credits 






■ 






■ 


Retention or degree completion at 
the same institution 














All institutions 






■ 






■ 


Four-year institutions 






■ 






■ 


Two-year institutions 














Retention or degree completion at 
any institution 














All institutions 






■ 






■ 


Four-year institutions 






■ 






■ 


Two-year institutions 






■ 






■ 


Bachelor’s degree or higher 






■ 






■ 


Associate’s degree or higher 














All institutions 






■ 






■ 


Two-year institutions 




■ 


■ 




■ 




Transfers to four-year institutions 






■ 






■ 



■=Positive and statistically significant; — =Negative and statistically significant. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service. National Evaluation of 
Student Support Services (SSS): Examination of Student Outcomes After Six Years, 2010. 



■ The average total amount of improvement from the HLM analysis that could be 

associated with the receipt of supplemental services (from any source) ranged from 12 
to 19 percentage points with regard to retention or degree completion, 8 to 10 
percentage points with regard to degree attainment, 16 percentage points with regard 
to transfers from two-year to four-year institutions, GPA increases of 0.14 on a 4.0 
scale, and an increase of 17 credits in the total credits earned over the six years. 
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Introduction 



This chapter starts the presentation of the research findings by examining the simplest 
method of modeling student participation in SSS: through a dichotomous (yes or no) measure of whether 
students participated in SSS . 20 Such an approach has both advantages and disadvantages. 

Advantages 

■ This method provides a single summary measure of the effects of SSS, and when 
multivariate linear regression is used, the applicable coefficient can be directly 
interpreted as the average overall effect among all SSS students. 

■ The “intent to treat” issue is resolved by treating all SSS participants in the same way, 
regardless of whether some students participated more or in different ways than 
others. Thus, if there are confounding factors such as motivation or academic 
neediness that affect students’ level of participation, those factors will not be allowed 
to give some SSS students greater weight than others. 

Disadvantages 

m This approach treats dosage as unimportant, as if all students are expected to 

experience the same effect regardless of which services they received or how much 
they received. 

■ Because only a single measure of SSS participation is used, and because the 
corresponding measures of alternative services are also dichotomous and highly 
interrelated with the receipt of SSS services, the statistical ability to distinguish the 
effects of SSS from the effects of other services is greatly lessened. Often it may only 
be possible to draw a conclusion about supplemental services in general, rather than to 
specifically estimate the association between SSS participation and student outcomes. 

■ The estimates of the SSS effect shown here may be underestimates because of a 
failure to account for the potential indirect effects of SSS through its effects on 
persistence after the second year. This is a greater issue when using dichotomous 
measures than it would be with more refined measures. 

Limitations 

■ This approach to the “intent to treat” issue does not eliminate the need to statistically 
adjust for such factors as motivation and academic neediness. Because the SSS 



20 Though this is the simplest method of modeling SSS participation, it is not the simplest method overall. The simplest approach 
would be a comparison of the mean outcomes for SSS and non-SSS outcomes. However, because the SSS students were more 
disadvantaged than the comparison group, such results would be likely to be misleading. All of the statistics presented here are 
based on multivariate models in which student and institution characteristics (such as academic background, demographic 
characteristics, and student attitudes) are included to adjust for the differences between the two groups. 
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students tended to be more disadvantaged than non-SSS students (and possibly may 
also have differed in motivation as shown by their choice to participate), it is 
necessary to adjust for such characteristics anyway. 

The dichotomous measures were examined using two different statistical techniques: (1) 
multivariate and logistic regression, and (2) hierarchical linear modeling. 



Multivariate and Logistic Regressions 



Exhibits 3-2 and 3-3 present the findings using the dichotomous approach, with exhibit 3-2 
showing the regression coefficients or log-odds ratios, and exhibit 3-3 showing the estimated effects 
across all SSS students. In general, the exhibits show that the receipt of supplemental services was 
associated with improved student outcomes for every academic outcome but one (the one exception was 
retention at the same institution for students starting at two-year institutions). However, the findings were 
much more limited with regard to first-year SSS participation in particular. First-year SSS participation 
was associated with improvements in grade point averages and receipt of an associate's degree or higher 
among students at two-year institutions but not with improvements on any of the other outcome measures. 
Because the three measures of supplemental services arc highly interrelated, it is possible that some of the 
effects of first-year SSS participation may be captured through the other variables. Also, the measure of 
services after the first year does include SSS services as well as non-SSS services, so any effect of SSS 
participation after the first year would be included within this more general measure of participation. The 
following improvements were identified as being associated with the receipt of supplemental services: 

■ Grade point averages. SSS students’ cumulative GPAs over six years were 
increased from an estimated 2.11 (if they had received no supplemental services) to an 
actual 2.34, with 0.06 of that increase being associated with first-year SSS 
participation and the remainder with supplemental services received after the first 
year. 

■ Total credits earned. The total number of credits earned for six years was increased 
from an estimated mean of 53.5 credits (if no supplemental services were received) to 
an actual mean of 75.5. None of that increase could be specifically associated with 
first-year SSS participation. 

■ Retention or baccalaureate degree completion. SSS students were 12 to 18 
percentage points more likely to be either still in college during the sixth year or to 
have received a baccalaureate degree than would be estimated if they had not received 
services. (However, no statistically significant improvements were found for 
retention or degree completion at the same institution for students starting at two-year 
institutions, possibly in part because of the reduced number of cases available for 
analysis.) For example, 22 percent are estimated to have remained at the same 
institution or completed a degree if they had not received supplemental services, 
compared with an actual 34 percent. None of the increases could be specifically 
associated with first-year SSS participation. 
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Exhibit 3-2. Regression coefficients concerning effects of SSS and non-SSS supplemental services, 



by outcome measure: Six-year outcomes after freshman entry in 199] 


L— 92 


Outcome measure 


SSS participation in 
first year 


Non-SSS supplemental 
services 


Services after the 
first year 


Cumulative (six-year) GPA 


0.063** 


— 


0.178** 


Cumulative (six-year) credits earned 


— 


— 


21.967** 


Retention or degree completion (combined) 1 








Same institution 








All institutions 


— 


— 


2.271** 


Four-year institutions only 


— 


— 


2.523** 


Two-year institutions only 


— 


— 


— 


Any institution 








All institutions 


— 


— 


3.090** 


Four-year institutions only 


— 


— 


4.074** 


Two-year institutions only 


— 


— 


3.861** 


Degree attainment 1 








Bachelor's degree or higher 


— 


— 


2.135** 


Associate’s degree or higher 








All institutions 


— 


-- 


1.814** 


Two-year institutions only 


— 


1.603** 


1.594* 


Transfers from two-year to four-year institutions 1 


— 


— 


4.177** 



— Not statistically significant. 

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown. 

1 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of 
completing a degree) can be expressed as an odds (e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio 
expresses the improvement in the probability that is associated with participation in SSS (e.g., if the odds ratio is 2, then doubling the odds of 1.5 
would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(100-75)=75/25=3). A log-odds ratio that is less than 1 
reflects a negative association between the receipt of supplemental services and the outcome indicated. 

NOTE: This table presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative 
models and alternative measures of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 

Examination of Student Outcomes After Six Years, 2010. 
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Exhibit 3-3. Estimated effects of SSS and non-SSS supplemental services on SSS students, by 
outcome measure: Six-year outcomes after freshman entry in 1991-92 



Outcome measure 


Estimated 
outcome without 
supplemental 
services 


Improvement 
associated with 
SSS participation 
in first year 


Improvement 
associated with 
other services 


Observed 

outcome 


Cumulative (six-year) GPA 


2.11 


0.063 


0.178 


2.34 


Cumulative (six-year) credits earned 


53.52 


— 


21.967 


75.49 


Retention or degree completion (combined) 










Same institution 










All institutions 


22% 


— 


12% 


34% 


Four-year institutions only 


26% 


— 


16% 


42% 


Two-year institutions only 


8% 


— 


0% 


8% 


Any institution 










All institutions 


47% 


— 


16% 


63% 


Four-year institutions only 


52% 


— 


18% 


70% 


Two-year institutions only 


19% 


— 


16% 


35% 


Degree attainment 










Bachelor's degree or higher 


28% 


— 


10% 


38% 


Associate’s degree or higher 










All institutions 


41% 


— 


8% 


49% 


Two-year institutions only 


23% 


7% 


6% 


36% 


Transfers from two-year to four-year 
institutions 


8% 


- 


11% 


19% 



—Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the 
required p-value. 



NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may 
not sum to the total effect, due to the transformation used to estimate probabilities based on the logistic regression coefficients. This table 
presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative models and alternative 
measures of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 
Examination of Student Outcomes After Six Years, 2010. 



■ Degree completion. SSS students were 8 to 13 percentage points more likely to have 
received either a bachelor’s degree or higher or associate’s degree or higher than if 
they had not received supplemental services. For example, 28 percent are estimated to 
have received a bachelor’s degree or higher if they had received no supplemental 
services, compared with 38 percent as the actual rate. None of the increases could be 
specifically associated with first-year SSS participation. 

■ Transfers from two-year to four-year institutions. The rate of transfer from two- 
year to four-year institutions was increased from an estimated 8 percent (if no services 
had been received) to an actual 19 percent. None of that increase was specifically 
associated with first-year SSS participation. 

Because the three types of supplemental services are so highly interrelated when measured through 
dichotomous variables, it is difficult to differentiate among them. One cannot be confident either that the 
two findings of positive and statistically significant relationships with first-year participation measure 
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only SSS effects, or that the other measures exclude SSS effects. The safest conclusion is that 
supplemental services in general appear to be associated with improved student outcomes. 

Hierarchical Linear Modeling 



As discussed in chapter 2, the variance estimates for the statistics in exhibit 3-1 may be 
incorrect due to a failure to account for the nesting of students within institutions, possibly leading to 
incorrect conclusions about the statistical significance of the findings. To examine this possibility, the 
regressions in exhibit 3-2 were rerun using HLM. Exhibit 3-4 presents the regression coefficients from 
the HLM runs, and exhibit 3-5 presents the average estimated effect on the various academic outcomes. 



Exhibit 3-4, HLM regression coefficients concerning effects of SSS and non-SSS supplemental 
services, by outcome six years after freshman entry in 1991-92 



Outcome measure 


SSS participation in 
first year 


Non-SSS supplemental 
services 


Services after the 
first year 


Cumulative (six-year) GPA 


— 


— 


0.177** 


Cumulative (six-year) credits earned 


— 


— 


22.206** 


Retention or degree completion (combined) 








Same institution 








All institutions 


— 


— 


0.151** 


Four-year institutions only 


— 


— 


0.198** 


Two-year institutions only 


— 


— 


— 


Any institution 








All institutions 


— 


— 


0.229** 


Four-year institutions only 


— 


— 


0.213** 


Two-year institutions only 


— 


— 


0.255** 


Degree attainment 








Bachelor's degree or higher 


— 




0.128** 


Associate’s degree or higher 








All institutions 


— 


-- 


0.103* 


Two-year institutions only 


— 


0.093* 


— 


Transfers from two-year to four-year institutions 


— 


— 


0.212** 



-- Not statistically significant. 

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown. 

NOTE: This table presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative 
models and alternative measures of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 
Examination of Student Outcomes After Six Years, 2010. 
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Exhibit 3-5. Estimated effects of SSS and non-SSS supplemental services on SSS students, by 
outcome six years after freshman entry in 1991-92 (HLM analysis) 



Outcome measure 


Estimated 
outcome without 
supplemental 
services 


Improvement 
associated with 
SSS participation 


Improvement 
associated with 
other services 


Observed 

outcome 


Cumulative (six-year) GPA 


2.20 


— 


0.136 


2.34 


Cumulative (six-year) credits earned 


58.50 


— 


16.991 


75.49 


Retention or degree completion (combined) 










Same institution 










All institutions 


22% 


— 


12% 


34% 


Four-year institutions only 


26% 


— 


16% 


42% 


Two-year institutions only 


8% 


— 


0% 


8% 


Any institution 










All institutions 


45% 


— 


18% 


63% 


Four-year institutions only 


53% 


— 


17% 


70% 


Two-year institutions only 


19% 


— 


16% 


35% 


Degree attainment 










Bachelor's degree or higher 


28% 


— 


10% 


38% 


Associate’s degree or higher 










All institutions 


41% 


— 


8% 


49% 


Two-year institutions only 


28% 


— 


8% 


36% 


Transfers from two-year to four-year 
institutions 


6% 


- 


13% 


19% 



—Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the 
required p-value. 



NOTE: This table presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative 
models and alternative measures of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 
Examination of Student Outcomes After Six Years, 2010. 



The HLM results indicate that no statistically significant improvements can specifically be 
associated with SSS participation on any of the student outcomes. Still, despite the lack of positive 
findings concerning SSS, the HLM results do indicate that supplemental services are associated with 
improved student outcomes: on two measures with regard to non-SSS services received in the first year 
(receiving an associate’s degree, when measured either across all institutions or only among two-year 
institutions) and on 10 measures with regal'd to services received after the first year. Note, however, that 
because of the broadness of the dichotomous measures and the high degree to which they are interrelated, 
the findings should be interpreted as providing general support on the benefits of supplemental services 
rather than as indicating that one form of supplemental services is superior to another. Following are the 
specific estimates of the amount of improvement in outcomes that was associated with receiving 
supplemental services: 

■ Grade point averages. SSS students’ cumulative GPAs over six years were 
increased from an estimated 2.20 (if they had received no supplemental services) to an 
actual 2.34. None of that increase could be specifically attributed to first-year 
participation in SSS. 

■ Total credits earned. The total number of credits earned for six years was increased 
from an estimated mean of 58.5 credits (if no supplemental services were received) to 
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an actual mean of 75.5. None of that increase could be specifically associated with 
first-year SSS participation. 

■ Retention or baccalaureate degree completion. SSS students were 12 to 18 
percentage points more likely to be either still in college during the sixth year or to 
have received a baccalaureate degree than would be estimated if they had not received 
services. (The one exception is that no statistically significant improvements were 
found for retention or degree completion at the same institution for students starting at 
two-year institutions, possibly because of the reduced number of cases available for 
analysis.) For example, 22 percent are estimated to have remained at the same 
institution or completed a degree if they had not received supplemental services, 
compared with an actual 34 percent. None of the increases could be specifically 
attributed to first-year participation in SSS. 

■ Degree completion. SSS students were 8 to 10 percentage points more likely to have 
received either a bachelor’s degree or higher or associate’s degree or higher than if 
they had not received supplemental services. For example, 28 percent are estimated to 
have received a bachelor’s degree or higher if they had received no supplemental 
services, compared with 38 percent as the actual rate. None of the increases were 
specifically associated with first-year SSS participation. 

■ Transfers from two-year to four-year institutions. The rate of transfer from two- 
year to four-year institutions was increased from an estimated 6 percent (if no services 
had been received) to an actual 19 percent. None of that increase was specifically 
associated with first-year SSS participation. 



Summary 



Though some minor differences appeared when using HLM rather than with multivariate 
and logistic regressions, both approaches largely produced similar results. The HLM results did not show 
positive and statistically significant associations between SSS participation and any of the 12 outcome 
measures, while the regression results showed such associations for only one of the 12 outcome measures. 
In general, the HLM estimates should be considered superior because they better account for the 
clustering of students within institutions. Both HLM and the regression results show positive and 
statistically significant associations for the same 1 1 academic outcomes when using general measures of 
supplemental services, and failed to show such associations for the remaining outcomes (i.e., retention or 
degree institution at the same institutions for students at two-year institutions only). The size of the 
estimated effect was highly similar using both approaches, with the total effect of all supplemental 
services typically varying by only 1 or 2 percentage points. 

Both statistical approaches also present the same difficulty with regard to drawing 
conclusions about the effects of SSS. Because of the substantial overlap among students in receiving all 
three types of supplemental services (SSS services during the first year, non-SSS services received during 
the first year, and SSS or non-SSS services received during later years), some effects of SSS may be 
included within the measures of supplemental services. Also, the measure of services received after the 
first year (which is the measure that most consistently showed positive and statistically significant 
relationships) includes SSS services as well as non-SSS services. Thus, the results cannot properly be 
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interpreted as indicating that SSS services do not show any effect while other supplemental services do. 
Rather, the best interpretation based on these results is that supplemental services in general have a 
positive effect, and we need more precise measures that better differentiate among the various types of 
services in order to estimate the specific contribution of each. 
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4. Research Findings Based on Continuous Measures 

of Participation 



Highlights of Findings 



■ Exhibit 4-1 summarizes the major results from the statistics presented in this chapter. 
There were consistent and statistically significant improvements in students’ academic 
outcomes if they received supplemental services. Unlike when dichotomous measures 
of participation were used, academic improvements often were specifically associated 
with SSS participation. The HLM analysis, which better accounts for the clustering of 
students in institutions, showed effects on cumulative GPA, retention or degree 
completion at the same institution, retention or degree completion at any institution, 
and degree attainment. 21 

Exhibit 4-1. Summary of findings for each outcome using continuous measures 



of participation, by method used: Six-year outcomes after freshman 
entry in 1991-92 



Outcome measure 


Regressions 


HLM 


First-year services 


Later 

services 


First-year services 


Later 

services 


SSS 


Other 


SSS 


Other 


GPA 


■ 


■ 


■ 


■ 


■ 


■ 


Credits 


■ 


■ 


■ 


• 


■ 


■ 


Retention or degree completion at 
the same institution 














All institutions 


■ 




■ 


■ 




■ 


Four-year institutions 


■ 


o 


■ 


■ 


o 


■ 


Two-year institutions 


• 




■ 


• 


■ 


■ 


Retention or degree completion at 
any institution 














All institutions 


■ 




■ 


■ 




■ 


Four-year institutions 


■ 


■ 


■ 


■ 


• 


■ 


Two-year institutions 


■ 




■ 






■ 


Bachelor’s degree or higher 


■ 




■ 


• 




■ 


Associate’s degree or higher 














All institutions 


• 




■ 


• 




■ 


Two-year institutions 






■ 






■ 


Transfers to four-year institutions 




• 


■ 




• 


■ 



■ Significant both with and without propensity scores; • Significant only without propensity scores; 
o Significant only with propensity scores. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support 
Services (SSS): Examination of Student Outcomes After Six Years, 2010. 



■ Using the HLM estimates, the average total amount of improvement that could be 

associated with the receipt of supplemental services (from any source) ranged from 15 
to 24 percentage points with regal'd to retention or degree completion across all 
institutions, 11 to 13 percentage points with regard to degree attainment across all 
institutions, 10 percentage points with regard to transfers from two-year to four-year 



21 All of the estimates concerning employment outcomes should be considered preliminary because the time period may have 
been too short to properly reflect the effect of education on employment. 
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institutions, GPA increases of 0.16 to 0.17 on a 4.0 scale, and an increase of 16 to 18 
credits in the total credits earned over the six years. 

■ Using the HLM estimates, first-year participation in SSS was correlated with the 
following improvements in student outcomes: an increase of 0.02 to 0.04 in the 
cumulative GPA, and increases of 4 to 9 percentage points in the likelihood of either 
remaining in college or completing a baccalaureate degree or higher across all 
institutions combined. For degree attainment, the HLM models were split on whether 
or not first-year participation in SSS was related to improvements in outcomes: 
models that included propensity scores showed no effect, while models that excluded 
propensity scores show improvements of 1 to 2 percent in degree attainment across all 
institutions. First-year participation did not show any association with improvements 
in transfers from two-year to four-year institutions. 



Introduction 



Chapter 3 provided evidence that the receipt of supplemental services is associated with 
improved academic outcomes but at best only weak evidence concerning the effect of SSS. Still, the 
dichotomous measures of participation that were used in chapter 3 were not well designed to differentiate 
between SSS participation and the receipt of other supplemental services: once participation in 
supplemental services was known, no additional benefit could be found that was associated with 
participating in SSS services, perhaps because most or all of the SSS benefit was already captured 
through the other measures. In this chapter, the extent and nature of SSS participation are measured more 
precisely through the use of nine distinct measures of SSS services, each based on the number of hours of 
services received in that area. The nine types of SSS services examined are instructional courses that are 
specially designated for SSS students, professional tutoring, peer tutoring, professional counseling, peer 
counseling, labs, workshops, cultural events, and services for the disabled. 

In addition, two dichotomous measures describing the SSS programs are also used: whether 
the programs were home-based, and whether the SSS services were blended with other services. 

Similarly, just as first-year SSS participation is modeled through multiple variables, the receipt of other 
supplemental services is also modeled through multiple variables, though in most cases (except for some 
of the measures of tutoring and counseling) participation is measured through a dichotomous variable 
rather than quantifying the extent of participation. The use of multiple, continuous measures of SSS 
participation has both advantages and disadvantages. 

Advantages 

■ The use of multiple, continuous measures provides a better capacity than the original 
dichotomous measures to differentiate between SSS services and other supplemental 
services. For example, while 91 percent of the SSS students received services outside 
of SSS, the correlations would tend to be much lower if one related a particular SSS 
service to a particular non-SSS service. Thus, there is a greater potential for 
measuring the effects specifically of SSS services rather than of supplemental services 
in general. 
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■ The resulting regression coefficients represent the improvements that are associated 
with specific services and thus provide a means of determining whether some services 
might especially be helpful. 

■ The statistical models are more realistic in that they take account of dosage, so that 
students who receive more services might be found to have different levels of 
improvement than students who receive few services. 

Disadvantages 

■ It is more difficult to determine the average effects across all SSS students because 
there is no single summary measure of participation that applies to all students. 
Because no service was received universally by all SSS students, any particular 
regression coefficient applies only to a subset of SSS students rather than to all of 
them. Because a student may receive more than one service, determining the total 
effect may also require summing multiple effects from individual services. Further, 
the total effect also depends on the amount of services received, so the regression 
coefficient must be multiplied by the amount of services rather than directly providing 
the average benefit from receiving the service. 

■ Because the amount of services received is being modeled, there is the potential that 
confounding factors (such as motivation or academic neediness) may be associated 
with both the level of services and students’ academic outcomes, possibly creating 
false associations between the receipt of services and student outcomes. To explore 
the importance of this possibility and to adjust the estimates appropriately, the 
statistical models were estimated both with and without the use of separate propensity 
measures that were created to measure students’ likelihood of receiving services. In 
principle, the propensity measures should capture the importance of students’ 
tendency to receive services, while the measures of participation would then only be 
associated with any additional change in student outcomes beyond that change that is 
associated with the propensity to receive services. 

Limitations 

■ It continues to be important to statistically adjust for such factors as motivation and 
academic neediness. Because the SSS students tended to be more disadvantaged than 
non-SSS students (and possibly may also have differed in motivation as shown by 
their choice to participate), it is necessary to adjust for such characteristics even when 
propensity measures are added to the models. 

■ The estimates of the SSS effect shown here still may be underestimates because of a 
failure to account for the potential indirect effect of SSS through its effect on 
persistence after the second year. Though this is a greater issue when using the 
dichotomous measures in chapter 3 than with the more refined measures used here, it 
remains a risk. 

Part of the structural design of this report is to use multiple research methodologies in order 
to allow readers to evaluate the implications of the various methodologies that might be applied. Thus, 
like chapter 3, this chapter presents the results of both multivariate and logistic regression, and 
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hierarchical linear modeling — which should provide the most accurate estimates because HLM is better 
able to account for the clustering of students within institutions. In addition to presenting these two 
methodological alternatives, this chapter presents the results both with and without propensity scores 
added to the model. Propensity scores are used here to estimate the number of hours of a particular type 
of service that a student would take if that option were available (i.e., if the student were in SSS and the 
service were offered at that institution). 22 

The purpose of including the propensity scores is to separate a student’ s general tendency to 
receive a service from the actual effect of that service, because the tendency itself could be correlated 
with student outcomes (e.g., a high level of motivation might be associated with improved student 
outcomes, and a high level of academic need might be associated with lower student outcomes). 
However, determining the value of including propensity scores is more difficult than determining the 
value of using HLM. Clearly it is necessary in some way to account for factors such as students’ 
motivation and academic need, and otherwise the measures of participation may be conflated with these 
other factors. Still, it is arguable whether it is sufficient to adjust for these factors through the addition of 
variables measuring these student characteristics, or whether it is also necessary to adjust for these 
characteristics by adding propensity scores to the model. Both types of adjustments have similar 
purposes, so their simultaneous use may not be necessary. One reason for providing statistics both with 
and without propensity measures is to allow researchers to evaluate these changes to the statistical 
models. 



Combining both of these methodological variations, the chapter is organized using the 
following structure, starting with the simplest models and advancing to the most complex: 



• Multivariate and logistic regressions 

- Without propensity scores 

- With propensity scores 

• Hierarchical linear modeling 

- Without propensity scores 

- With propensity scores 

This chapter primarily seeks to describe the results for each of the four models, and compare 
how they are different, without coming to a conclusion about which approach provides the best estimates. 
A final synthesis of these results and those from chapter 3 is provided in chapter 5. 

Multivariate and Logistic Regressions 



This section presents the research results when using multivariate and logistic regressions. 
As indicated, multivariate and logistic regressions are here used in two ways: without the use of 
propensity scores, and with their use. 



22 Propensity scores are also discussed in chapter 2 as part of the discussion on research methodology. 
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Without propensity scores 



Exhibits 4-2 and 4-3 present the findings using multiple measures of participation and no 
separate measures of propensity to receive services, with exhibit 4-2 showing the regression coefficients 
or log-odds ratios and exhibit 4-3, the estimated effects across all SSS students. To simplify the 
presentation of the effect estimates, and because ED’s primary interest is in the net effect of SSS and 
other supplemental services, only the net effects are presented in exhibit 4-3. (One still can determine 
which particular first-year SSS services showed positive and statistically significant relationships by 
examining exhibit 4-2.) 

These measures differ substantially in their performance from the dichotomous measures 
used in chapter 3 with regard both to finding effects of supplemental services in general (statistically 
significant improvements in performance appeared for all of the 12 measures, compared with 11 of the 12 
measures in chapter 3) and especially the ability to differentiate SSS effects from general effects (separate 
first-year SSS effects appeared for 11 of the 12 measures, compared with 1 of the 12 when using the 
dichotomous measures). The improved performance of the SSS participation measures is consistent with 
the increased capacity to differentiate among the different types of services, while the dichotomous 
models had little ability to distinguish between SSS and non-SSS services. 

The following improvements were identified as being associated with the receipt of 
supplemental services: 

■ Grade point averages. SSS students’ cumulative GPAs over six years were 
increased from an estimated 2.09 (if they had received no supplemental services) to an 
actual 2.34, with 0.12 of that increase being associated with first-year SSS 
participation and the remainder with other supplemental services. 

■ Total credits earned. The total number of credits earned for six years was increased 
from an estimated mean of 56.7 credits (if no supplemental services were received) to 
an actual mean of 75.5. First-year SSS participation was associated with 2.6 credits of 
that increase, while the remainder was associated with the receipt of other 
supplemental services. 

■ Retention or baccalaureate degree completion. SSS students were 13 to 21 
percentage points more likely (across all institutions) either to be still in college 
during the sixth year or to have received a baccalaureate degree than would be 
estimated if they had not received services. For example, 21 percent are estimated to 
have remained at the same institution or completed a degree if they had not received 
supplemental services, compared with an actual 34 percent. First-year SSS 
participation was associated with improvements of between 4 and 5 percentage points 
across all institutions. 

■ Degree completion. SSS students were 11 to 14 percentage points more likely 
(across all institutions) to have received either a bachelor’s degree or higher or 
associate's degree or higher than if they had not received supplemental services. For 
example, 24 percent are estimated to have received a bachelor’s degree or higher if 
they had received no supplemental services, compared with 38 percent as the actual 
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Exhibit 4-2. Regression coefficients concerning effects of SSS services using continuous measures of SSS participation and no 
measures of propensity to receive services, by outcome six years after freshman entry in 1991-92 



Outcome measure 


Instruc- 

tional 

courses 


Profes- 

sional 

tutoring 


Peer 

tutoring 


Profes- 

sional 

counseling 


Peer 

counseling 


Labs 


Workshops 


Cultural 

events 


Services 
for the 
disabled 


Home- 

based 


Blended 


Cumulative (six-year) GPA 


— 


— 


0.006** 


-0.019** 


— 


— 


— 


— 


— 


0.159** 


— 


Cumulative (six-year) credits 
earned 


- 


- 


0.281** 


-1.176** 


-4.581** 


- 


- 


1.763** 


- 


- 


- 


Retention or degree 
completion (combined) 1 
























Same institution 
























All institutions 


— 


— 


— 


0.929** 


— 


— 


1.044* 


— 


1.048** 


— 


1.454** 


Four-year institutions only 


— 


— 


— 


0.926** 


— 


— 


1.044* 


— 


— 


— 


1.466** 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


3.515* 


— 


Any institution 
























All institutions 


— 


— 


1.017** 


— 


— 


— 


— 


— 


— 


— 


1.649** 


Four-year institutions only 


— 


— 


1.025** 


0.923** 


— 


— 


— 


— 


— 


— 


1.581** 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


3.641** 


1.859* 


Degree attainment 1 
























Bachelor’s degree or higher 


1.004* 


— 


1.018** 


0.930** 


— 


0.945** 


— 


— 


— 


1.440** 


— 


Associate’s degree or higher 
























All institutions 


1.003* 


— 


1.013** 


0.941** 


— 


0.963** 


— 


— 


— 


— 


— 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


Transfers from two-year to 
four-year institutions 1 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 



— Not statistically significant. 

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown. 

1 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 
(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 
odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(100-75)=75/25=3). The estimated effect of SSS is based on 
observed retention rates at the same institution of 34 percent across all institutions, and 43 percent across two-year institutions. 

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 
services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 
This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 
of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010. 




Exhibit 4-3. Estimated effects of SSS and non-SSS supplemental services on SSS students using continuous measures of SSS 



participation and no measures of propensity to receive services, by outcome six years after freshman entry in 1991-92 



Outcome measure 


Estimated outcome 
without supplemental 
services 


Improvement associated 
with SSS participation in 
first year 


Improvement associated 
with other services 


Observed outcome 


Cumulative (six-year) GPA 


2.09 


0.121 


0.131 


2.34 


Cumulative (six-year) credits earned 


56.72 


2.601 


16.170 


75.49 


Retention or degree completion (combined) 










Same institution 










All institutions 


21% 


4% 


10% 


34% 


Four-year institutions only 


25% 


5% 


12% 


42% 


Two-year institutions only 


6% 


1% 


2% 


8% 


Any institution 










All institutions 


42% 


5% 


18% 


63% 


Four-year institutions only 


47% 


5% 


16% 


70% 


Two-year institutions only 


19% 


6% 


11% 


35% 


Degree attainment 










Bachelor’s degree or higher 


24% 


6% 


3% 


38% 


Associate’s degree or higher 










All institutions 


38% 


3% 


8% 


49% 


Two-year institutions only 


31% 


0% 


5% 


36% 


Transfers from two-year to four-year institutions 


12% 


0% 


7% 


19% 



NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 
transformation used to estimate probabilities based on the logistic regression coefficients. This table presents only the findings when continuous measures of SSS participation were used. See 
chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures of SSS. All effect estimates are based on regression coefficients that were statistically significant 
when using 0.05 as the required p- value. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 





rate. First-year SSS participation was associated with an improvement of 6 percentage 
points with regal'd to completing a bachelor’s degree or higher, 3 percentage points 
with regard to completing an associate’s degree or higher at any institution, and no 
improvement with regard to completing an associate’s degree or higher at two-year 
institutions. 

■ Transfers from two-year to four-year institutions. The rate of transfer from two- 
year to four-year institutions was increased from an estimated 12 percent (if no 
services had been received) to an actual 19 percent. None of that increase was 
specifically associated with first-year SSS participation. 



With propensity scores 

Repeating the same structure used previously, but now using models that include propensity 
scores, exhibits 4-4 and 4-5 present the findings using multiple measures of participation: Exhibit 4-4 
shows the regression coefficients or log-odds ratios, and exhibit 4-5 shows the estimated effects across all 
SSS students. Like exhibits 4-2 and 4-3, these measures show more a more consistent association 
between the receipt of supplemental services and students outcomes than the dichotomous measures used 
in chapter 3, with regard both to finding effects of supplemental services in general (statistically 
significant improvements in performance appeared for all of the 12 measures, compared with 11 of the 12 
measures in chapter 3) and the ability to differentiate SSS effects from general effects (separate first-year 
SSS effects appeared for 8 of the 12 measures, compared with 1 of the 12 when using the dichotomous 
measures). The results for first-year SSS participation were somewhat weaker with propensity scores 
than without them, with statistically significant improvements appearing for 8 of 12 measures (rather than 
11 of 12), and fewer SSS services showing positive and statistically significant relationships with student 
outcomes. 



The following improvements were identified as being associated with the receipt of 
supplemental services: 

■ Grade point averages. SSS students’ cumulative GPAs over six years were 
increased from an estimated 2.05 (if they had received no supplemental services) to an 
actual 2.34, with 0.14 of that increase being associated with first-year SSS 
participation and the remainder with other supplemental services. 

■ Total credits earned. The total number of credits earned for six years was increased 
from an estimated mean of 57.0 credits (if no supplemental services were received) to 
an actual mean of 75.5. First-year SSS participation was associated with 2.4 credits of 
that increase, while the remainder was associated with the receipt of other 
supplemental services. 

■ Retention or baccalaureate degree completion. SSS students were 2 to 28 
percentage points more likely either to be still in college during the sixth year or to 
have received a baccalaureate degree than would be estimated if they had not received 
services. For example, 18 percent are estimated to have remained at the same 
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Exhibit 4-4. Regression coefficients concerning the effects of SSS services using continuous measures of SSS participation and 



adding propensity scores, by outcome six years after i 


freshman entry in ] 


L991-92 


Outcome measure 


Instruc- 

tional 

courses 


Profes- 

sional 

tutoring 


Peer 

tutoring 


Profes- 

sional 

counseling 


Peer 

counseling 


Labs 


Workshops 


Cultural 

events 


Services 
for the 
disabled 


Home- 

based 


Blended 


Cumulative (six-year) GPA 


- 


- 


0.006** 


-0.036** 


-0.056* 


- 


- 


- 


- 


0.195** 


- 


Cumulative (six-year) credits 
earned 


— 


— 


0.366** 


-1.864* 


-4.098** 


— 


— 


— 


— 


— 


— 


Retention or degree completion 
(combined) 1 
























Same institution 
























All institutions 


- 


- 


- 


0.908* 


- 


- 


- 


- 


- 


1.558** 


1.369** 


Four-year institutions only 


- 


- 


- 


0.874* 


- 


- 


- 


- 


- 


1.687** 


1.356* 


Two-year institutions only 


- 


- 


- 


- 


- 


0.925* 


- 


- 


- 


- 


- 


Any institution 
























All institutions 


0.995* 


- 


- 


- 


- 


- 


- 


- 


- 


1.657** 


1.605** 


Four-year institutions only 


- 


- 


- 


0.816** 


- 


- 


- 


- 


- 


1.860** 


1.581** 


Two-year institutions only 


0.989* 


- 


- 


- 


- 


- 


- 


- 


- 


2.822* 


- 


Degree attainment 1 
























Bachelor’s degree or higher 














- 






1.655** 


- 


Associate’s degree or higher 
























All institutions 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


Two-year institutions only 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


Transfers from two-year to four- 
year institutions 1 


0.9834* 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 



— Not statistically significant. 

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown. 

1 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 
(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 
odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(1 00-75)=75/25=3). A log-odds ratio that is less than 1 reflects a 
negative association between the receipt of supplemental services and the outcome indicated. 

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 
services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 
This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 
of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010. 




Exhibit 4-5. Estimated effects of SSS and non-SSS supplemental services on SSS students using continuous measures of SSS 



participation and adding propensity scores, by outcome six years after freshman entry in 1991-92 



Outcome measure 


Estimated outcome 
without supplemental 
services 


Improvement associated 
with SSS participation in 
first year 


Improvement associated 
with other services 


Observed outcome 


Cumulative (six-year) GPA 


2.05 


0.143 


0.144 


2.34 


Cumulative (six-year) credits earned 


57.01 


2.386 


16.091 


75.49 


Retention or degree completion (combined) 










Same institution 










All institutions 


18% 


7% 


9% 


34% 


Four-year institutions only 


21% 


9% 


12% 


42% 


Two-year institutions only 


6% 


0% 


2% 


8% 


Any institution 










All institutions 


40% 


8% 


17% 


63% 


Four-year institutions only 


42% 


9% 


16% 


70% 


Two-year institutions only 


32% 


2% 


1% 


35% 


Degree attainment 










Bachelor’s degree or higher 


24% 


5% 


10% 


38% 


Associate’s degree or higher 










All institutions 


40% 


0% 


9% 


49% 


Two-year institutions only 


29% 


0% 


7% 


36% 


Transfers from two-year to four-year institutions 


11% 


0% 


8% 


19% 



NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 
transformation used to estimate probabilities based on the logistic regression coefficients. This table presents only the findings when continuous measures of SSS participation were used. See 
chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures of SSS. All effect estimates are based on regression coefficients that were statistically significant 
when using 0.05 as the required p- value. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 





institution or completed a degree if they had not received supplemental services, 
compared with an actual 34 percent. First-year SSS participation was associated with 
improvements of between 7 and 9 percentage points at four-year institutions and 
across all institutions, and of 0 to 2 percentage points at two-year institutions. 

■ Degree completion. SSS students were 7 to 14 percentage points more likely to have 
received either a bachelor’s degree or higher or associate's degree or higher than if 
they had not received supplemental services. For example, 24 percent are estimated to 
have received a bachelor’s degree or higher if they had received no supplemental 
services, compared with 38 percent as the actual rate. First-year SSS participation 
was associated with an improvement of 5 percentage points with regal’d to completing 
a bachelor’ s degree or higher, and no improvement with regard to completing an 
associate’s degree or higher. 

■ Transfers from two-year to four-year institutions. The rate of transfer from two- 
year to four-year institutions was increased from an estimated 1 1 percent (if no 
services had been received) to an actual 19 percent. None of that increase was 
specifically associated with first-year SSS participation. 



Hierarchical Linear Modeling 



As suggested earlier, HLM is better designed to estimate variances than ordinary least 
squares regression when students are clustered within institutions. This section repeats the analysis in the 
previous section of the report, but using HLM. 

Without propensity scores 

Exhibit 4-6 presents the regression coefficients from the HLM runs when no propensity 
scores are included in the HLM models, and exhibit 4-7 presents the average estimated effects on the 
various academic outcomes. These results are quite similar to the results in the previous section of this 
chapter, though one outcome (retention or degree completion at any institution for students at two-year 
institutions) did not show statistically significant relationships with the receipt of SSS. However, the 
results were very unlike the HLM results in chapter 3, with first-year SSS participation showing positive 
and statistically significant associations with 9 of the 12 outcomes (compared with 0 of 12 when using the 
dichotomous models). The specific findings were as follows: 

■ Grade point averages. SSS students’ cumulative GPAs over six years were 
increased from an estimated 2.17 (if they had received no supplemental services) to an 
actual 2.34. Of that increase, 0.04 was associated with first-year participation in SSS, 
and the remainder with the receipt of other supplemental services. 

■ Total credits earned. The total number of credits earned for six years was increased 
from an estimated mean of 56.3 credits (if no supplemental services were received) to 
an actual mean of 75.5. Three credits of the 19-credit increase were specifically 
associated with first-year SSS participation. 
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Exhibit 4-6. HLM regression coefficients concerning the effects of SSS services using continuous measures of SSS participation and 
no measures of propensity, by outcome six years after freshman entry in 1991-92 



Outcome measure 


Instruc- 

tional 

courses 


Prof. 

tutoring 


Peer 

tutoring 


Prof. 

counseling 


Peer 

counseling 


Labs 


Work- 

shops 


Cultural 

events 


Services 
for the 
disabled 


Home- 

based 


Blended 


Cumulative (six-year) GPA 


— 


— 


0.006** 


-0.014* 


— 


— 


— 


— 


— 


— 


— 


Cumulative (six-year) credits 
earned 


- 


- 


0.391** 


-0.887* 


- 


- 


- 


1.476** 


- 


- 


- 


Retention or degree completion 
(combined) 1 
























Same institution 
























All institutions 


— 


— 


0.003** 


-0.011** 


— 


— 


— 


— 


0.006** 


— 


0.064* 


Four-year institutions only 


— 


— 


0.003** 


-0.012** 


— 


— 


— 


— 


— 


— 


0.074* 


Two-year institutions only 


— 


-0.016** 


— 


— 


— 


— 


— 


— 


0.009** 


0.085* 


0.058* 


Any institution 
























All institutions 


— 


— 


0.003** 


— 


— 


— 


— 


— 


— 


0.077** 


0.065* 


Four-year institutions only 


— 


— 


0.003** 


— 


— 


— 


— 


— 


— 


0.059* 


0.069* 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 


-0.061** 


— 




— 


Degree attainment 1 
























Bachelor’s degree or higher 


— 


— 


0.002** 


-0.009* 


— 


— 


— 




— 


— 


— 


Associate’s degree or higher 














— 


— 


— 


— 


— 


All institutions 


— 


— 


0.002* 


-0.009* 


— 


-0.005** 


— 


— 


— 


— 


— 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 




— 


— 


— 


Transfers from two-year to four- 
year institutions 1 


- 


- 


- 


- 


- 


- 


-0.020* 


-0.039* 


- 


- 


- 



— Not statistically significant. 

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown. 

1 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 
(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 
odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(1 00-75)=75/25=3). A log-odds ratio that is less than 1 reflects a 
negative association between the receipt of supplemental services and the outcome indicated. 

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 
services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 
This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 
of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010. 





Exhibit 4-7. Estimated effects of SSS and non-SSS supplemental services on SSS students when using continuous measures and no 



propensity scores, by outcome six years after freshman entry in 1991-92 (HEM 


[ analysis) 


Outcome measure 


Estimated outcome 
without supplemental 
services 


Improvement associated 
with SSS participation in 
first year 


Improvement associated 
with other services 


Observed outcome 


Cumulative ( six-year ) GPA 


2.17 


0.039 


0.136 


2.34 


Cumulative ( six-year ) credits earned 


56.32 


3.191 


15.978 


75.49 


Retention or degree completion ( combined ) 










Same institution 










All institutions 


19% 


5% 


10% 


34% 


Four-year institutions only 


24% 


5% 


13% 


42% 


Two-year institutions only 


1% 


4% 


3% 


8% 


Any institution 










All institutions 


39% 


9% 


15% 


63% 


Four-year institutions only 


46% 


9% 


15% 


70% 


Two-year institutions only 


24% 


— 


11% 


35% 


Degree attainment 










Bachelor’s degree or higher 


25% 


2% 


11% 


38% 


Associate’s degree or higher 










All institutions 


37% 


1% 


11% 


49% 


Two-year institutions only 


28% 


— 


8% 


36% 


Transfers from two-year to four-year institutions 


9% 


— 


10% 


19% 



—Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the required p-value. 

NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 
transformation used to estimate probabilities based on the logistic regression coefficients. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 




■ Retention or baccalaureate degree completion. Overall, SSS students were 15 to 
24 percentage points more likely (across all institutions) either to be still in college 
during the sixth year or to have received a baccalaureate degree than would be 
estimated if they had not received services. For example, 19 percent are estimated to 
have remained at the same institution or completed a degree if they had not received 
supplemental services, compared with an actual 34 percent. Between 5 and 9 
percentage points of the increases were associated with first-year participation in SSS. 

■ Degree completion. Overall, SSS students were 12 to 13 percentage points more 
likely (across all institutions) to have received either a bachelor’s degree or higher or 
associate’s degree or higher than if they had not received supplemental services. For 
example, 25 percent are estimated to have received a bachelor’s degree or higher if 
they had received no supplemental services, compared with 38 percent as the actual 
rate. Between 1 and 2 percentage points of that increase were associated with first- 
year participation in SSS. 

■ Transfers from two-year to four-year institutions. The rate of transfer from two- 
year to four-year institutions was increased from an estimated 9 percent (if no services 
had been received) to an actual 19 percent. None of that increase was specifically 
associated with first-year SSS participation. 



With propensity scores 

Because of the potential that the level of participation in SSS might be confounded with 
factors such as motivation and academic need, the HLM models were also run with propensity scores 
included. Exhibit 4-8 presents the regression coefficients from the HLM runs, and exhibit 4-9 presents 
the average estimated effects on the various academic outcomes. 

Similar to the regression results, the HLM results show that the receipt of supplemental 
services was associated with improved student outcomes for every academic outcome that was tested. 
However, the results were weaker with regard to outcomes that could specifically be associated with first- 
year SSS participation: there was an association between such participation and improved outcomes with 
regard to cumulative GPA, and retention or degree completion (at four-year institutions and at all 
institutions combined), but no SSS effect was identified with regard to degree attainment (if examined 
alone and not in combination with retention), transfers from two-year to four-year institutions, and 
retention or degree completion at two-year institutions. The results with regard to first-year SSS 
participation were also weaker when compared with the HLM models that did not use propensity 
measures: they showed positive and statistically significant relationships to only 5 of the 12 outcomes, 
compared with 11 of 12 when no propensity scores were included, and typically only one SSS measure 
achieved statistical significance per outcome, rather than up to three measures as shown in exhibit 4-6. 
Still, these results were stronger than when dichotomous measures of participation were used (in chapter 
3), suggesting that the greater ability to differentiate among students was helpful in identifying the effects 
of services. 
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Exhibit 4-8. HLM regression coefficients concerning the effects of SSS services using continuous measures of SSS participation 
with propensity scores, by outcome six years after freshman entry in 1991-92 



Outcome measure 


Instruc- 

tional 

courses 


Prof. 

tutoring 


Peer 

tutoring 


Prof. 

counseling 


Peer 

counseling 


Labs 


Workshops 


Cultural 

events 


Services 
for the 
disabled 


Home- 

based 


Blended 


Cumulative (six-year) GPA 


— 


— 


— 


— 


— 


0.010* 


— 


— 


— 


— 


— 


Cumulative (six-year) credits 
earned 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


Retention or degree completion 
(combined) 1 
























Same institution 
























All institutions 


— 


— 


— 


— 


— 


— 


— 


— 


— 


0.063* 


0.060* 


Four-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


0.074* 


— 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


Any institution 
























All institutions 


— 


— 


— 


— 


— 


— 


— 


— 


— 


0.077** 


— 


Four-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


0.072* 


— 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


Degree attainment 1 
























Bachelor’s degree or higher 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


Associate’s degree or higher 
























All institutions 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


Two-year institutions only 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


— 


Transfers from two-year to four- 
year institutions 1 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 


- 



— Not statistically significant. 

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown. 

1 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 
(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 
odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(1 00-75)=75/25=3). A log-odds ratio that is less than 1 reflects a 
negative association between the receipt of supplemental services and the outcome indicated. 

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 
services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 
This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 
of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 





Exhibit 4-9. Estimated effects of SSS and non-SSS supplemental services on SSS students when using continuous measures and 



propensity scores, by outcome six years after freshman entry in 1991-92 (HLM 


[ analysis) 


Outcome measure 


Estimated outcome 
without supplemental 
services 


Improvement associated 
with SSS participation in 
first year 


Improvement associated 
with other services 


Observed outcome 


Cumulative (six-year) GPA 


2.18 


0.021 


0.137 


2.34 


Cumulative (six-year) credits earned 


59.22 


0.0 


16.265 


75.49 


Retention or degree completion (combined) 










Same institution 










All institutions 


17% 


6% 


11% 


34% 


Four-year institutions only 


24% 


5% 


13% 


42% 


Two-year institutions only 


6% 


0% 


2% 


8% 


Any institution 










All institutions 


44% 


4% 


15% 


63% 


Four-year institutions only 


51% 


4% 


15% 


70% 


Two-year institutions only 


24% 


0% 


11% 


35% 


Degree attainment 










Bachelor’s degree or higher 


27% 


0% 


11% 


38% 


Associate’s degree or higher 










All institutions 


38% 


0% 


11% 


49% 


Two-year institutions only 


28% 


0% 


8% 


36% 


Transfers from two-year to four-year institutions 


9% 


0% 


10% 


19% 



NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 
transformation used to estimate probabilities based on the logistic regression coefficients. This table presents only the findings when continuous measures of SSS participation were used. See 
chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures of SSS. All effect estimates are based on regression coefficients that were statistically significant 
when using 0.05 as the required p- value. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 




Following are the specific estimates of the amount of improvement in outcomes that was 
associated with receiving supplemental services: 

■ Grade point averages. SSS students’ cumulative GPAs over six years were 
increased from an estimated 2.18 (if they had received no supplemental services) to an 
actual 2.34. Of that increase, 0.02 was specifically associated with first-year 
participation in SSS. 

■ Total credits earned. The total number of credits earned for six years was increased 
from an estimated mean of 59.2 credits (if no supplemental services were received) to 
an actual mean of 75.5. None of that increase could be specifically associated with 
first-year SSS participation. 

■ Retention or baccalaureate degree completion. SSS students were 17 to 19 
percentage points more likely (across all institutions) either to be still in college 
during the sixth year or to have received a baccalaureate degree than would be 
estimated if they had not received services. Between 4 and 6 percentage points of that 
increase was specifically associated with first-year SSS participation. 

■ Degree completion. SSS students were 8 to 11 percentage points more likely to have 
received either a bachelor’s degree or higher or associate’s degree or higher than if 
they had not received supplemental services. For example, 27 percent are estimated to 
have received a bachelor’s degree or higher if they had received no supplemental 
services, compared with 38 percent as the actual rate. None of the increases were 
specifically associated with first-year SSS participation. 

■ Transfers from two-year to four-year institutions. The rate of transfer from two- 
year to four-year institutions was increased from an estimated 9 percent (if no services 
had been received) to an actual 19 percent. None of that increase was specifically 
associated with first-year SSS participation. 



Types of Services 



Because the measures of supplemental services in this chapter were tied to specific services, 
it is possible to examine whether some types of services were especially likely to be effective. Exhibit 4- 
10 shows which specific first-year SSS services were found to have positive effects for each outcome 
measure. Similarly, because the focus is on types of services, and a given type of service presumably 
could be effective whether offered through SSS or through other sources, exhibit 4-11 shows which other 
supplemental services were found to have positive effects for each outcome measure. For brevity, only 
those variables that were positive and statistically significant are shown. 23 Also for brevity, only the 
findings from the HLM analyses are included, because these models should have the most reliable 
standard errors and thus provide the best test of whether a service had a significant effect. However, 
because deciding whether or not to include propensity scores is a less straightforward decision, the 
exhibits cover both methodologies, and show whether the statistical significance was based on either or 
both methodologies. 



23 Statistical significance is based on the 0.05 level. 
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Some services stood out as being statistically significant both with and without the use of 

24 

propensity scores:" 



■ Home-based SSS programs (for two outcome measures); 

■ Blended programs (for one outcome measure); 

■ College re-entrance counseling (for one outcome measure); 

■ Cultural enrichment (outside of SSS) in the first year (for two outcome measures) or 

cultural trips in later years (for 10 outcome measures); 

■ Referrals to resources in the first year (for one outcome measure); 

■ Counseling in 1993-94 (for 9 outcome measures), or in later years (for 11 outcome 

measures); 

■ Tutoring in 1992-93 (for one outcome measure) or 1993-94 (for 7 outcome 
measures); or 

■ Recent contacts with supplemental services (for 1 1 outcome measures). 

A few other services were statistically significant either one way or the other, but not both: 



Significant only without the use of propensity measures: 

■ SSS peer tutoring 

■ SSS cultural events in the first year 

■ SSS services for the disabled in the first year 

■ Non-SSS services in the first year for students with limited English ability 

Significant only with the use of propensity measures: 

■ SSS labs in the first year 

■ Non-SSS services for the disabled in the first year 

A few other patterns also are apparent in exhibits 4-10 and 4-11: 

■ Services received after the first year appeal - to be significantly related to long-term 
student outcomes much more frequently than services received in the first year. 

■ Among those services received in the first year, SSS services more frequently were 
statistically significant that non-SSS services. 



24 The counts provided here are the counts of outcome measures that showed statistical significance both with and without the use 
of propensity measures. In addition, some of the variables listed below also were significant for other outcome measures using 
only one approach (i.e., either with or without the use of propensity measures but not for both for the same outcome measure). 



58 




VO 



Exhibit 4-10. First-year SSS services that showed positive and statistically significant effects on student outcomes, by outcome six 
years after freshman entry in 1991-92 



Outcome measure 


Peer tutoring 


Labs 


Cultural events 


Services for the 
disabled 


Home-based 


Blended 


Cumulative (six-year) GPA 


• 


o 










Cumulative (six-year) credits earned 


• 




• 








Retention or degree completion (combined) 














Same institution 














All institutions 


• 






• 


o 


■ 


Four-year institutions only 


• 








o 


• 


Two-year institutions only 








• 


• 


• 


Any institution 














All institutions 


• 








■ 


• 


Four-year institutions only 


• 








■ 


• 


Two-year institutions only 














Degree attainment 














Bachelor’s degree or higher 


• 












Associate’s degree or higher 














All institutions 


• 












Two-year institutions only 














Transfers from two-year to four-year institutions 















■ Significant both with and without propensity scores; • Significant only without propensity scores; o Significant only with propensity scores. 

NOTE: Statistical significance is based on the 0.05 level. This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary 
in chapter 5 for the findings when using dichotomous measures of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 





Exhibit 4-11. Other supplemental services that showed positive and statistically significant effects on student outcomes, by outcome 
six years after freshman entry in 1991-92 



Outcome measure 


Non-SSS first-year services 


Later-year services 


Services 
for the 
disabled 


Limited 

English 

ability 


College re- 
entrance 
counseling 


Cultural 

enrich- 

ment 


Referrals 

to 

resources 


Counsel- 
ing in 
1992 


Counsel- 
ing in 
1993 


Tutor- 
ing in 
1992 


Tutor- 
ing in 
1993 


Recent 

counsel- 

ing 


Recent 

trip 


Recent 

contact 


Cumulative (six-year) GPA 








■ 


■ 




■ 






■ 


■ 


■ 


Cumulative (six-year) 
credits earned 








■ 






■ 


■ 


■ 


■ 


■ 


■ 


Retention or degree 
completion (combined) 


























Same institution 


























All institutions 














■ 


o 


■ 


■ 


■ 


■ 


Four-year institutions 
only 


o 












■ 




■ 


■ 


■ 


■ 


Two-year institutions 
only 
























■ 


Any institution 


























All institutions 














■ 




■ 


■ 


■ 


■ 


Four-year institutions 
only 




• 














■ 


■ 


■ 


■ 


Two-year institutions 
only 






■ 












■ 


■ 


■ 


■ 


Degree attainment 


























Bachelor’s degree or 
higher 














■ 






■ 


■ 


■ 


Associate’s degree or 
higher 


























All institutions 














■ 






■ 


■ 


■ 


Two-year institutions 
only 














■ 






■ 




■ 


Transfers from two-year to 
four-year institutions 






• 








■ 




■ 


■ 


■ 





■ Significant both with and without propensity scores; • Significant only without propensity scores; o Significant only with propensity scores; 

NOTE: Statistical significance is based on the 0.05 level. This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary 
in chapter 5 for the findings when using dichotomous measures of SSS. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 





■ SSS peer tutoring in the first year is unique among first-year services in reaching 
statistical significance for 8 of the 12 measures, although only when propensity scores 
were not used. 

■ Home-based and blended SSS programs were the other SSS characteristics that were 
statistically significant for more than two student outcomes. 



Summary 



All of the models considered in this chapter models showed statistically significant 
improvements for the 12 student outcomes if students had received supplemental services. The measures 
of SSS participation, which were limited to the first year, performed less well than many of the other 
measures of supplemental services. This is consistent with a general pattern that the relationships were 
stronger for services received after the first year than for first-year services. (Students’ participation in 
SSS was not limited to the first year, but the data do not discriminate between SSS and non-SSS services 
after the first year.) Still, depending on the methodological approach that was used, between 5 and 10 
student outcomes were positively related to first-year SSS participation. 

The regression and HLM models differed primarily by the HLM models being less likely to 
show statistically significant relationships for first-year SSS participation: fewer outcomes showed such 
relationships, and fewer SSS services had positive and statistically significant coefficients. 

Fundamentally, however, the results were not greatly different. Perhaps the greatest difference was that 
the negative but statistically significant results that appeared in the regression models largely disappeared 
in the HLM models. 

Much greater differences appeared when comparing the models with and without propensity 
measures. The models that excluded propensity measures showed a greater variety of first-year SSS 
services as being positive and statistically significant, and across a greater number of outcomes. In fact 
the models that included propensity measures primarily indicated the importance of home-based programs 
for selected student outcomes, rather than showing a mixture of services as affecting student outcomes. 
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5. Summary and Conclusions 

This chapter compares the findings from the four types of approaches used in chapters 3 and 
4 to determine which findings are consistently supported regardless of the approach, which findings 
depend on the particular approach being used, and the importance of the alternative methodological 
approaches. It then presents an overall conclusion about which findings (from our perspective) best 
describe the association between participation in SSS and student outcomes, and the implications of the 
findings for the SSS program. 

Overview 

The results varied depending on whether one focuses specifically on first-year SSS services 
or on supplemental services in general (exhibit 5-1). With regard to first-year SSS services, none of the 
12 outcome measures showed consistent positive findings across all six methodologies. This was 
primarily due to the poor performance of the SSS measures when using the dichotomous measures of 
participation, because none of the dichotomous HLM models and only one of the dichotomous regression 
models showed positive effects from SSS. The poor performance of the dichotomous models was 
expected because of the weaknesses of the dichotomous measures of participation. The remaining four 
models showed consistent positive effects for five outcome measures, mixed results for five measures, 
and no significant effects for two measures. However, even the models using continuous measures of 
SSS participation performed less well with regard to first-year SSS participation than they did for 
supplemental services in general. This was most noticeable when the propensity scores were included, 
with the HLM findings showing positive effects for 5 of the 12 student outcomes and the regression 
results showing positive effects for 8 of the 12 outcomes. First-year SSS participation did better when the 
propensity scores were excluded (i.e., with positive outcomes for 9 of the 12 outcomes in the HLM results 
and 10 of the 12 outcomes in the regression results), though still less well than other supplemental 
services (11 of 12 outcomes). Also, though the differences were often small, the regression results tended 
to result in higher effects for SSS services than did the HLM results (13 times versus 5 times in which the 
HLM effects were greater). 
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Exhibit 5-1. Comparison of estimated effects of first-year SSS services using each of six methodologies: Six-year outcomes after 
freshman entry in 1991-92 



Outcome measure 


Dichotomous measures of services 


Continuous measures of services 


Six-year 

outcome 


No propensity scores 


With propensity scores 


Regressions 


FILM 


Regressions 


HLM 


Regressions 


HLM 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Cumulative ( six-year ) GPA (A=4.0) 


2.28 


0.06 


2.34 


— 


2.22 


0.12 


2.3 


0.04 


2.2 


0.14 


2.32 


0.02 


2.34 


Cumulative ( six-year ) credits earned 


75.49 


— 


75.49 


— 


72.89 


2.60 


72.3 


3.19 


73.1 


2.39 


75.49 


— 


75.49 


Retention or degree completion 
{ combined ) 




























Same institution 




























All institutions 


34% 


— 


34% 


— 


30% 


4% 


29% 


5% 


27% 


7% 


28% 


6% 


34% 


Four-year institutions only 


42% 


— 


42% 


— 


37% 


5% 


37% 


5% 


33% 


9% 


37% 


5% 


42% 


Two-year institutions only 


8% 


— 


8% 


— 


7% 


1% 


4% 


4% 


8% 


— 


8% 


— 


8% 


Any institution 




























All institutions 


63% 


— 


63% 


— 


58% 


5% 


54% 


9% 


55% 


8% 


59% 


4% 


63% 


Four-year institutions only 


70% 


— 


70% 


— 


65% 


5% 


61% 


9% 


61% 


9% 


66% 


4% 


70% 


Two-year institutions only 


35% 


— 


35% 


— 


29% 


6% 


35% 


— 


33% 


2% 


35% 


— 


35% 


Degree attainment 




























Bachelor’s degree or higher 


38% 


— 


38% 


— 


32% 


6% 


36% 


2% 


33% 


5% 


38% 


— 


38% 


Associate’s degree or higher 




























All institutions 


49% 


— 


49% 


— 


46% 


3% 


48% 


1% 


49% 


— 


49% 


— 


49% 


Two-year institutions only 


36% 


— 


36% 


— 


36% 


— 


36% 


— 


36% 


— 


36% 


— 


36% 


Transfers from two-year to four-year 
institutions 


19% 




19% 




19% 




19% 




19% 




19% 




19% 



— Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the required p- value. 

NOTE: The column labeled start shows the estimated outcome if the SSS students had not participated in SSS but had still received the same other supplemental services as in the study, while 
the gain shows the estimated improvement according to the statistical model that was used. Note that the starting level is a statistical estimate, calculated by subtracting the estimated 
improvement from the actual (six-year) outcome. For this reason, the starting level varies depending on the statistical model that was used. The six-year outcome is the mean outcome for all 
applicable SSS participants who were in the study, as measured through their transcripts and self-reports. For example, the mean six-year cumulative GPA among all SSS participants in the 
study was 2.34, and, among all SSS students in the study who started as freshmen in two-year institutions, 19 percent transferred to a four-year institution at some time during the six years of 
the study. Shaded cells indicate positive and statistically significant improvements in student outcomes. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 





The results were stronger and more consistent when looking at supplemental services in 
general (exhibit 5-2). Often the results were quite similar across all six alternative methodologies. For 11 
outcomes, all six methodologies showed significant improvement in outcomes, while they were split on 
one outcome. For the only outcome for which there were mixed results, it was the dichotomous models 
that failed to find significant improvements, while the continuous models did find significant 
improvements. The effect estimates tended to fall within a small range and generally could not be 
considered as being significantly different from each other. Counting even the smallest differences in 
order to detect overall patterns, the FILM results produced higher estimates of effects 15 times and the 
regression results did so 14 times. The models that did not include propensity scores tended to produce 
higher estimates of effects slightly more often (10 times) than the models with propensity scores (eight 
times), but this difference is too small to be meaningful. The continuous measures producing higher 
estimates of effects 32 times and the dichotomous measures producing higher estimates 13 times. 

When comparing exhibits 5-1 and 5-2, it is clear that there is a stronger case that 
supplemental services make a difference than that SSS services do. Several alternative explanations 
might be offered for such a difference. 

■ Non-SSS services may be more effective. However, there is no clear reason for such a 
difference. Since institutions need to submit plans for their SSS programs to win 
grants, and since the grants are awarded competitively, it seems likely that they are 
administered in a competent manner; if anything one might expect that they would 
perform better rather than doing more poorly than other supplemental services. 

■ SSS services may be hindered by federal requirements for the programs. Again, such 
an explanation is theoretically possible but seems premature without evidence of such 
hindrances, especially given that the programs have an extra source of funds (i.e., 
federal funding for SSS). 

■ Later-year services may be more effective than first-year services, especially with 
regard to long-term outcomes. This explanation seems more likely than the previous 
two, and in fact some researchers might be surprised to see first-year services still 
showing an effect in the sixth year. In fact, this could be a reason why SSS program 
characteristics (home-based programs, and blended programs) in some models seem 
to be more important than measures of specific first-year services; these general 
program characteristics may be acting as surrogates for SSS participation over 
multiple years. 

■ The models may not be fully differentiating between SSS and non-SSS services, so that 
some of the effect of SSS services is captured through the alternative measures. 

• One reason is that any SSS services received after the first year are included 
within the alternative measures (because of an inability to determine the 
source of the services), rather than being treated as SSS services. In fact, it is 
possible that because the comparison group was chosen to have comparable 
disadvantages as the SSS students, non-SSS students in the study may have 
qualified for and received SSS services in later years, so even the comparison 
group may have benefited from SSS services after the first year. 
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Exhibit 5-2. Comparison of estimated effects of supplemental services using each of six methodologies: Six-year outcomes after 
freshman entry in 1991-92 



Outcome measure 


Dichotomous measures of services 


Continuous measures of services 


Six-year 

outcome 


No propensity scores 


With propensity scores 


Regressions 


HLM 


Regressions 


HLM 


Regressions 


HLM 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Start 


Gain 


Cumulative ( six-year ) GPA (A=4.0) 


2.11 


0.23 


2.20 


0.14 


2.09 


0.25 


2.17 


0.17 


2.05 


0.29 


2.18 


0.16 


2.34 


Cumulative ( six-year ) credits earned 


53.52 


21.97 


58.50 


16.99 


56.72 


18.77 


56.32 


19.17 


57.01 


18.48 


59.22 


16.27 


75.49 


Retention or degree completion 
{ combined ) 




























Same institution 




























All institutions 


22% 


12% 


22% 


12% 


21% 


13% 


19% 


15% 


18% 


16% 


17% 


17% 


34% 


Four-year institutions only 


26% 


16% 


26% 


16% 


25% 


17% 


24% 


18% 


21% 


21% 


24% 


18% 


42% 


Two-year institutions only 


8% 


— 


8% 


— 


6% 


2% 


1% 


7% 


6% 


2% 


6% 


2% 


8% 


Any institution 




























All institutions 


47% 


16% 


45% 


18% 


42% 


21% 


39% 


24% 


40% 


23% 


44% 


19% 


63% 


Four-year institutions only 


52% 


18% 


53% 


17% 


47% 


23% 


46% 


24% 


42% 


28% 


51% 


19% 


70% 


Two-year institutions only 


19% 


16% 


19% 


16% 


19% 


16% 


24% 


11% 


22% 


13% 


24% 


11% 


35% 


Degree attainment 




























Bachelor’s degree or higher 


28% 


10% 


28% 


10% 


24% 


14% 


25% 


13% 


24% 


14% 


27% 


11% 


38% 


Associate’s degree or higher 




























All institutions 


41% 


8% 


41% 


8% 


38% 


11% 


37% 


12% 


40% 


9% 


38% 


11% 


49% 


Two-year institutions only 


23% 


13% 


28% 


8% 


31% 


5% 


28% 


8% 


29% 


7% 


28% 


8% 


36% 


Transfers from two-year to four-year 
institutions 


8% 


11% 


6% 


13% 


12% 


7% 


9% 


10% 


11% 


8% 


9% 


10% 


19% 



— Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the required p- value. 

NOTE: The column labeled start shows the estimated outcome if the SSS students had not participated in supplemental services, while the gain shows the estimated improvement according to 
the statistical model that was used. Note that the starting level is a statistical estimate, calculated by subtracting the estimated improvement from the actual (six-year) outcome. For this reason, 
the starting level varies depending on the statistical model that was used. The six-year outcome is the mean outcome for all applicable SSS participants who were in the study, as measured 
through their transcripts and self-reports. For example, the mean six-year cumulative GPA among all SSS participants in the study was 2.34, and, among all SSS students in the study who 
started as freshmen in two-year institutions, 19 percent transferred to a four-year institution at some time during the six years of the study. Shaded cells indicate positive and statistically 
significant improvements in student outcomes. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 
2010. 





• With the dichotomous models, the overlap between receiving SSS 

services and other services was so great that apparently little variation 
was left to explain after the other supplemental services had been 
modeled. Though the better differentiation between services with the 
use of multiple continuous measures should have lessened this problem 
(and in fact these models did show a greater effect for SSS when 
compared with the dichotomous models), it may not have eliminated it 
completely. It may be that the most critical question is whether a 
student is a recipient of a package of services, and that the measures of 
later-year services and first-year non-SSS services are largely 
sufficient to identify students receiving such packages. In that case, 
there may be little variation left to be explained by the first-year SSS 
variables. 

■ The inclusion of a measure of whether students had attended school some time 
after the first year could result in underestimating the effect of SSS. This 
danger was noted in chapter 2. However, the inclusion of this variable might 
have similar effects on the other measures of supplemental services, and thus it 
is not clear why this should result in weaker findings for SSS than for the other 
measures of supplemental services. 

■ The use of propensity scores may have contributed to underestimating the 
effects of first-year SSS services. In the models without propensity scores, the 
first-year SSS services were more likely to be significant than when the 
propensity scores were included, though they were still less likely to show 
significant relationships than some of the measures of recent services. This 
issue is discussed in greater detail later in this chapter. 

Of all of these explanations, the third and fourth appear to provide the most credible 
explanations for the weaker findings concerning SSS than for supplemental services in general: 
later-year services may be more critical with regard to affecting long-term outcomes, or the models 
may be failing to fully differentiate between SSS and non-SSS services, so that some of the effect 
of SSS is captured through the other measures. The last explanation concerning the use of 
propensity scores might also help to partially explain the weaker findings for SSS, though it would 
not provide a complete explanation because the results were still somewhat weaker even when 
propensity scores were excluded. 

Choosing Among the Six Models 

Though the models are largely consistent in finding positive effects from 
supplemental services in general, and more limited positive effects from first-year SSS 
participation in particular, the differences in the models raise the question as to which estimates 
should be considered the most reliable. 

■ One part of the answer should be relatively straightforward: the HLM 
estimates should generally be considered to be superior to the regression 
estimates because HLM is better designed for accounting for the clustering of 
students within institutions. The greatest handicap of HLM is that no 
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accommodation was made for the fact that some outcome variables were 
dichotomous, because HLM did not converge to a solution when using this 
option. In this regal'd, it is reassuring that the logistic regressions (which did 
account for the limited dependent variables) often produced similar results. 

■ Choosing between the dichotomous measures and continuous measures may be 
more difficult. The advantage of the dichotomous measures is that one does 
not risk introducing other factors (such as academic need and motivation) that 
may be correlated with students’ levels of participation. On the other hand, 
such a large proportion of SSS students also received supplemental services 
from other sources that the dichotomous models are likely to underestimate the 
effects of SSS and, instead, absorb some of that effect within the other 
measures of supplemental services. The continuous measures, through their 
greater level of detail, are better designed to discriminate between SSS and 
other services, while the inclusion of propensity scores or a variety of measures 
of student characteristics should help to statistically adjust for any bias 
associated with conflating participation with academic need or motivation. 
Thus, the continuous measures seem to be the best designed for measuring the 
effects of SSS. 

■ Perhaps the most difficult choice is the decision whether to include or not 
include propensity measures. The argument for including propensity measures 
is that students’ level of involvement in SSS may be interrelated with factors 
such as motivation and academic need; thus, there is a risk that the statistical 
models might measure the importance of these factors with regard to student 
outcomes rather than giving accurate measures of the effects of SSS. The 
purpose of adding propensity measures is to model the likelihood of 
participation separately, so that the SSS participation measures will only 
capture the added benefit from participating in SSS, distinct from these other 
factors. This argument is an important one, and there clearly is a need to 
somehow account for the importance of these other factors. On the other hand, 
there are also arguments against the need for or benefit from using propensity 
measures. 

• Factors such as motivation and academic need are included in the 
statistical models in any case, and the regression equations should 
be better able to individually fit those variables when they are 
measured as multiple variables in the context of a particular student 
outcome, rather than through standardized formulas that are the 
same for every model. 

• While the propensity scores were derived individually (i.e., one 
score at a time), they are used in combination. The reason for using 
nine separate propensity measures is that participation in one 
service showed little relation to participation in another. On the 
other hand, when all nine are used together in a single model, they 
may interact in a way that might confuse the measurement process. 
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• If irrelevant variables are added to a regression correlation and they 
are correlated with other variables of importance, their inclusion 
may result in increased standard errors that may interfere with 
examining those other variables. The propensity scores do show a 
fairly high correlation with the measures of first-year SSS services 
(ranging from 0.22 for services for disabled students to 0.64 for 
peer tutoring), so this is a real possibility. The facts that peer 
tutoring especially had a high correlation and that the results for 
peer tutoring were the most sensitive to the inclusion of propensity 
scores (i.e., peer tutoring was the SSS variable that performed the 
best when propensity scores were excluded but that never was 
statistically significant when including propensity scores) are 
consistent with this explanation. 

• As one test of whether the propensity scores belonged in the model, 
few if any of the propensity scores were statistically significant. 

For six of the 12 outcome measures, no propensity score was 
statistically significant, while one was significant for four outcome 
measures, two were significant for one outcome measures, and three 
were significant for one outcome measure. The propensity scores 
were retained under the principle that they were theoretically 
necessarily. However, an alternative approach is to consider the 
other measures of student characteristics as also adjusting for 
students’ propensity to participate, in which case it would be 
reasonable to drop those propensity scores that were not significant. 
Exhibit 5-3 shows which SSS services would be considered to have 
positive and statistically significant effects in that case. 

• The actual performance of the propensity measures is in some ways 
counterintuitive. The SSS variable that stood out most strongly was 
home-based programs, while the actual services generally were not 
statistically significant when propensity scores were included. Yet 
even if the organization of the programs is important, presumably 
the actual effect comes from the receipt of supplemental services; 
one would not expect a student who officially belonged to SSS but 
never received any SSS services to experience much benefit. 25 
There is at least one possible explanation why the variable for 
home-based programs may be capturing all of the first-year effects: 
it is possible that it acts as a surrogate for receiving a full package 
of services (because one of the purposes of a home-based program 
is to see that students receive such a package), and that there is little 
or no extra benefit to be measured once the effect of the “package” 
is measured. Alternatively, the absence of significant results for 
individual services but not home -based programs could be 



25 It is possible that simply belonging to SSS may help to create some identification with the institution and with other 
students, so home-based services may provide some benefit that extends beyond the receipt of services. Still, it seems 
unlikely that all of the benefit from SSS is independent of the receipt of services. 
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considered as a sign that the statistical model was not performing 
well. 



Weighing all these factors together, the best choice appears to be to include 
propensity scores only when they are statistically significant. 



Policy Findings Concerning SSS 

The single largest finding is that both supplemental services in general and first-year 
SSS services in particular have a positive effect on students’ academic outcomes, though SSS 
effects were not detected for as many outcome measures as supplemental services in general. This 
section reviews which particular outcomes were affected by SSS or by supplemental services in 
general. It also examines what implications these findings have for how SSS might best be 
administered. More specifically, these findings involve the relative importance of specific SSS 
services and the importance of receiving services beyond the first year. 

Exhibit 5-3. First-year SSS services that showed positive and statistically significant effects 



on student outcomes when only statistically significant propensity scores are 
retained, by outcomes six years after freshman entry in 1991-92 



Outcome measure 


Peer tutoring 


Services for 
the disabled 


Home-based 


Blended 


Cumulative (six-year) GPA 










Cumulative (six-year) credits earned 










Retention or degree completion (combined) 










Same institution 










All institutions 




✓ 






Four-year institutions only 










Two-year institutions only 






✓ 


V 


Any institution 










All institutions 






✓ 




Four-year institutions only 






✓ 




Two-year institutions only 










Degree attainment 










Bachelor's degree or higher 










Associate’s degree or higher 










All institutions 










two-year institutions only 










Transfers from two-year to four-year institutions 











S Statistically significant at 0.05 level. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 



Types of outcomes affected by SSS. Based on the hypothesis that the HLM models 
with continuous measures of SSS services provide the most reliable measures of effect, and that 
propensity scores are used only when statistically significant, SSS appeared to have a positive 
effect on seven measures of student outcomes, while the remaining five outcomes showed no 
effect. However, supplemental services in general appeared to have positive effects on all 12 
outcomes, providing some justification for inferring that SSS might be helpful in those areas as 
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well. Following are the complete findings, together with some speculations about why some 
outcome measures showed no SSS effects. 

■ GPAs. Both first-year SSS services and other supplemental services were 
shown to have a positive effect on students’ cumulative GPAs. 

■ Credits earned. No effect from first-year SSS participation was found on the 
total credits earned, though other supplemental services did have an effect. 

■ Retention or degree completion (combined measure). First-year SSS 
participation had a positive effect on retention or degree completion for every 
measure but retention or degree completion at any institution (for two-year 
institutions only). Other supplemental services were found to have a positive 
effect for the remaining measure (as well as on the measures where first- year 
SSS participation showed a positive association with improved student 
outcomes). 

■ Degree attainment (alone). First-year SSS participation was positively 
related to earning an associate’s degree or higher at all institutions but not to 
the other two measures of degree attainment. Supplemental services were 
found to have a positive effect on all three measures. Possibly the weaker 
findings for degree attainment alone as compared with degree attainment when 
combined with retention are because the six-year time span may not be 
adequate for many disadvantaged students to complete a degree. As shown in 
exhibit 2-4, 25 percent of SSS students were still in college though they had 
not yet attained a baccalaureate degree. 

■ Transfers from two-year to four-year institutions. No specific effect from 
first-year SSS participation was detected, though supplemental services were 
found to have a positive effect. All models that were limited only to two-year 
institutions tended to perform poorly, very likely due to the limited number of 
cases available for analysis. 

The importance of specific SSS services. The specific SSS services that were found 
to have a positive effect were home-based programs, blended programs, peer tutoring, and services 
for the disabled. Other services that had a positive effect, and that may or may not have been 
offered through SSS, were counseling, field trips or cultural enrichment, referrals to outside 
resources, services for the disabled and for those with limited English ability, college reentrance 
counseling, and recent contacts with support services. 

One should be wary of judging that any remaining services failed to have an effect. 
From a statistical viewpoint, the failure to find a statistically significant difference only means that 
the difference could have occurred by chance, but not that it did. Further, some of the findings, 
including the performance of the measures of home -based programs, referrals to other services, 
and recent contacts with support services, may suggest both that it is important that students 
receive a package of services and that the contributions of specific services may not appear 
independently but rather be incorporated within indicators that students received such a package. 
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The importance of services beyond the first year. As noted elsewhere, some of the 
services that most consistently were statistically significant were those received after the first year, 
and especially in the most recent years. No data are available on whether or not those services 
were offered through SSS. However, these findings suggest that SSS should not be viewed simply 
as a first-year program but that the continued provision of resources is an important tool in 
improving student outcomes. 
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Appendix A: Detailed Tables 



NOTE: This appendix provides the complete regression models used in the analysis. For brevity, 
because of the large number of models that were tested, only the results of the HLM analysis arc 
shown (under the assumption that they provide more accurate estimates of the standard errors). 
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Exhibit A-l. HLM analysis to predict cumulative GPAs six years after 



freshman entry in 1991-92 using dichotomous measures of 
participation 



V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


2.4102 


0.0246 


98.0840 


0.0000 


Two-year institution 


0.0695 


0.0656 


1.0600 


0.2960 


Doctorate -granting institution 


-0.0721 


0.0538 


-1.3400 


0.1870 


Age in 1991 


0.0273 


0.0027 


10.0700 


0.0000 


Black student 


-0.3321 


0.0354 


-9.3730 


0.0000 


Hispanic student 


-0.1301 


0.0450 


-2.8900 


0.0040 


American Indian student 


-0.2584 


0.0800 


-3.2310 


0.0020 


Asian student 


-0.0733 


0.0577 


-1.2720 


0.2040 


Female student 


0.1410 


0.0238 


5.9170 


0.0000 


High school GPA 


0.2944 


0.0274 


10.7410 


0.0000 


Percentile on SAT/ACT 


0.2387 


0.0558 


4.2790 


0.0000 


Took any developmental course 


-0.1558 


0.0238 


-6.5370 


0.0000 


Hours on school activities 


0.0036 


0.0006 


6.1760 


0.0000 


Attended college after first year 


0.6618 


0.0577 


11.4650 


0.0000 


No parental education beyond high school 


0.0259 


0.0203 


1.2710 


0.2040 


Family income greater than $20,000 


0.0248 


0.0246 


1.0110 


0.3120 


Above average academic ability 


0.1408 


0.0193 


7.3030 


0.0000 


Would attend free tutoring 


-0.0779 


0.0255 


-3.0520 


0.0030 


Have major concerns about ed. finances 


-0.0375 


0.0181 


-2.0750 


0.0380 


Once I start something, I finish it 


0.0467 


0.0190 


2.4540 


0.0140 


Feel comfortable on this campus 


0.0597 


0.0253 


2.3590 


0.0180 


Participated in SSS in first year 


0.0282 


0.0339 


0.8300 


0.4110 


Received non-SSS services in first year 


0.0186 


0.0292 


0.6370 


0.5240 


Received services after first year 


0.1775 


0.0362 


4.9060 


0.0000 


Missing data on student age 


0.6446 


0.0790 


8.1570 


0.0000 


Missing data on student race 


0.0432 


0.0741 


0.5840 


0.5590 


Missing data on high school GPA 


0.7922 


0.1186 


6.6770 


0.0000 


Missing data on SAT/ACT 


0.0259 


0.0409 


0.6330 


0.5260 


Missing data on developmental courses 


-0.2461 


0.0657 


-3.7490 


0.0000 


Missing data for hours on school activities 


0.1413 


0.0425 


3.3230 


0.0010 


Missing data on first generation 


-0.0922 


0.0576 


-1.6000 


0.1090 


Missing data on income 


-0.0362 


0.0350 


-1.0340 


0.3020 


Missing data on perceived ability 


0.1391 


0.1084 


1.2830 


0.2000 


Missing data on attending free tutoring 


-0.0864 


0.1661 


-0.5200 


0.6020 


Missing data on financial concerns 


-0.0562 


0.1084 


-0.5190 


0.6040 


Missing data on finishing tasks 


0.1792 


0.1206 


1.4850 


0.1370 


Missing data on comfort on campus 


-0.1804 


0.1848 


-0.9760 


0.3300 


Missing data on services in later years 


0.2412 


0.0896 


2.6920 


0.0080 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-2. HLM analysis to predict the number of credits earned six 



years after freshman entry in 1991-92 using dichotomous 
measures of participation 



V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


76.1699 


1.7481 


43.5730 


0.0000 


Two-year institution 


-20.0641 


3.5524 


-5.6480 


0.0000 


Doctorate -granting institution 


4.5546 


5.0934 


0.8940 


0.3760 


Black student 


-6.2885 


1.4786 


-4.2530 


0.0000 


Hispanic student 


0.8139 


1.8453 


0.4410 


0.6590 


American Indian student 


-18.5136 


3.6791 


-5.0320 


0.0000 


Asian student 


5.1328 


2.9719 


1.7270 


0.0840 


Female student 


3.1824 


1.4458 


2.2010 


0.0280 


Student was full-time 


14.0263 


3.1303 


4.4810 


0.0000 


High school GPA 


10.5841 


1.4056 


7.5300 


0.0000 


Percentile on SAT/ACT 


10.0523 


4.0474 


2.4840 


0.0130 


Took any developmental course 


-6.3317 


1.6473 


-3.8440 


0.0000 


Hours on school activities 


0.2194 


0.0381 


5.7560 


0.0000 


Attended college after first year 
No parental education beyond high 


41.0143 


2.5731 


15.9400 


0.0000 


school 


-1.2227 


1.2949 


-0.9440 


0.3450 


Family income greater than $20,000 


-1.7380 


1.6204 


-1.0730 


0.2840 


Above average academic ability 


2.5914 


1.4739 


1.7580 


0.0780 


Have major concerns about ed. finances 


-3.6605 


1.3501 


-2.7110 


0.0070 


Once I start something, I finish it 


4.5540 


1.4359 


3.1720 


0.0020 


Feel comfortable on this campus 


7.2484 


1.9053 


3.8040 


0.0000 


Participated in SSS in first year 


3.6342 


2.3208 


1.5660 


0.1170 


Received non-SSS services in first year 


0.6991 


1.2667 


0.5520 


0.5810 


Received services after first year 


22.2065 


1.9249 


11.5360 


0.0000 


Missing data on student race 


-2.2149 


4.4667 


-0.4960 


0.6200 


Missing data on full-time status 


25.4795 


7.2609 


3.5090 


0.0010 


Missing data on high school GPA 


29.2348 


5.2492 


5.5690 


0.0000 


Missing data on SAT/ACT 


-2.3667 


1.9763 


-1.1980 


0.2310 


Missing data on developmental courses 
Missing data for hours on school 


-6.9558 


4.8878 


-1.4230 


0.1550 


activities 


6.0046 


3.5506 


1.6910 


0.0900 


Missing data on first generation 


-3.2014 


2.5629 


-1.2490 


0.2120 


Missing data on income 


-2.4127 


1.7562 


-1.3740 


0.1700 


Missing data on perceived ability 


1.4748 


5.2493 


0.2810 


0.7790 


Missing data on financial concerns 


-9.8968 


4.4124 


-2.2430 


0.0250 


Missing data on finishing tasks 


9.5735 


7.2156 


1.3270 


0.1850 


Missing data on comfort on campus 


-1.0909 


8.4039 


-0.1300 


0.8970 


Missing data on services in later years 


1.7242 


5.2594 


0.3280 


0.7430 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-3. HLM analysis to predict retention or baccalaureate degree 



completion at the same institution six years after freshman entry in 
1991-92 using dichotomous measures of participation 



V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.3188 


0.0111 


28.6890 


0.0000 


Two-year institution 


-0.2671 


0.0254 


-10.5100 


0.0000 


Doctorate -granting institution 


0.0550 


0.0290 


1.8960 


0.0640 


Female student 


0.0260 


0.0138 


1.8850 


0.0590 


Student was full time 


0.0463 


0.0260 


1.7790 


0.0750 


High school GPA 


0.0669 


0.0129 


5.1790 


0.0000 


Percentile on SAT/ACT 


0.1297 


0.0435 


2.9840 


0.0030 


Took any developmental course 


-0.0533 


0.0160 


-3.3360 


0.0010 


Hours on school activities 


0.0010 


0.0003 


3.0730 


0.0030 


Attended college after first year 
No parental education beyond high 


0.1462 


0.0254 


5.7620 


0.0000 


school 


-0.0006 


0.0145 


-0.0390 


0.9690 


Family income greater than $20,000 


-0.0196 


0.0158 


-1.2370 


0.2160 


Once I start something, I finish it 


0.0313 


0.0133 


2.3520 


0.0190 


Feel comfortable on this campus 


0.0622 


0.0181 


3.4380 


0.0010 


Participated in SSS in first year 


0.0180 


0.0222 


0.8110 


0.4170 


Received non-SSS services in first year 


-0.0071 


0.0125 


-0.5720 


0.5670 


Received services after first year 


0.1505 


0.0216 


6.9710 


0.0000 


Missing data on full-time status 


0.0886 


0.0739 


1.1990 


0.2310 


Missing data on high school GPA 


0.2175 


0.0454 


4.7870 


0.0000 


Missing data on SAT/ACT 


0.0209 


0.0205 


1.0190 


0.3090 


Missing data on developmental courses 
Missing data for hours on school 


-0.1219 


0.0294 


-4.1460 


0.0000 


activities 


0.0065 


0.0309 


0.2100 


0.8340 


Missing data on first generation 


0.0146 


0.0291 


0.5020 


0.6150 


Missing data on income 


-0.0233 


0.0209 


-1.1170 


0.2650 


Missing data on finishing tasks 


0.0135 


0.0850 


0.1580 


0.8750 


Missing data on comfort on campus 


-0.0003 


0.0892 


-0.0030 


0.9970 


Missing data on services in later years 


0.0723 


0.0386 


1.8710 


0.0610 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-4. HLM analysis to predict retention or baccalaureate degree 



completion at any institution six years after freshman entry in 
1991-92 using dichotomous measures of participation 



V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.6187 


0.0115 


53.8190 


0.0000 


Two-year institution 


-0.2134 


0.0280 


-7.6180 


0.0000 


Doctorate -granting institution 


0.0485 


0.0292 


1.6600 


0.1040 


Black student 


0.0211 


0.0189 


1.1210 


0.2630 


Hispanic student 


0.0622 


0.0237 


2.6250 


0.0090 


American Indian student 


-0.1069 


0.0567 


-1.8860 


0.0590 


Asian student 


0.0967 


0.0342 


2.8250 


0.0050 


Female student 


0.0453 


0.0141 


3.2080 


0.0020 


Newly married 


-0.1065 


0.0284 


-3.7430 


0.0000 


High school GPA 


0.0530 


0.0121 


4.3800 


0.0000 


Percentile on SAT/ACT 


0.0876 


0.0339 


2.5810 


0.0100 


Took any developmental course 


-0.0561 


0.0146 


-3.8470 


0.0000 


Hours on school activities 


0.0012 


0.0004 


2.6610 


0.0080 


Attended college after first year 
No parental education beyond high 


0.3285 


0.0301 


10.9110 


0.0000 


school 


-0.0568 


0.0146 


-3.8940 


0.0000 


Family income greater than $20,000 


0.0032 


0.0152 


0.2140 


0.8310 


Above average academic ability 


0.0258 


0.0139 


1.8500 


0.0640 


Participated in SSS in first year 


0.0202 


0.0158 


1.2760 


0.2020 


Received non-SSS services in first year 


-0.0029 


0.0160 


-0.1800 


0.8580 


Received services after first year 


0.2292 


0.0228 


10.0660 


0.0000 


Missing data on student race 


0.0986 


0.0526 


1.8760 


0.0600 


Missing data on full-time status 


-0.0258 


0.0722 


-0.3570 


0.7210 


Missing data on high school GPA 


0.1514 


0.0486 


3.1150 


0.0020 


Missing data on SAT/ACT 


-0.0438 


0.0212 


-2.0700 


0.0380 


Missing data on developmental courses 
Missing data for hours on school 


-0.0568 


0.0333 


-1.7070 


0.0870 


activities 


-0.0011 


0.0408 


-0.0270 


0.9790 


Missing data on first generation 


-0.0522 


0.0359 


-1.4560 


0.1450 


Missing data on income 


-0.0234 


0.0229 


-1.0200 


0.3080 


Missing data on perceived ability 


0.0350 


0.0587 


0.5970 


0.5500 


Missing data on services in later years 


0.0700 


0.0511 


1.3680 


0.1710 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-5. HLM analysis to predict baccalaureate degree completion (or 

higher) at any institution six years after freshman entry in 1991-92 



using dichotomous measures of participations 


Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.3988 


0.0148 


27.0210 


0.0000 


Two-year institution 


-0.2272 


0.0309 


-7.3620 


0.0000 


Doctorate-granting institution 


0.0654 


0.0369 


1.7700 


0.0830 


Black student 


-0.0369 


0.0241 


-1.5300 


0.1260 


Hispanic student 


0.0041 


0.0221 


0.1840 


0.8540 


American Indian student 


-0.1613 


0.0542 


-2.9730 


0.0030 


Asian student 


0.0929 


0.0285 


3.2660 


0.0010 


Female student 


0.0309 


0.0156 


1.9750 


0.0480 


Live in college housing or frat./sorority 


0.0703 


0.0190 


3.6990 


0.0000 


Student was full-time 


0.0597 


0.0272 


2.2000 


0.0280 


Newly married 


-0.0715 


0.0298 


-2.4000 


0.0170 


High school GPA 


0.0962 


0.0145 


6.6330 


0.0000 


Percentile on SAT/ACT 


0.1246 


0.0315 


3.9540 


0.0000 


Took any developmental course 


-0.0904 


0.0163 


-5.5320 


0.0000 


Hours on school activities 


0.0017 


0.0005 


3.1120 


0.0020 


Attended college after first year 


0.1574 


0.0335 


4.7040 


0.0000 


No parental education beyond high school 


-0.0256 


0.0150 


-1.7060 


0.0880 


Family income greater than $20,000 


0.0203 


0.0141 


1.4320 


0.1520 


Above average academic ability 


0.0419 


0.0134 


3.1360 


0.0020 


Have major concerns about ed. finances 


-0.0398 


0.0121 


-3.2770 


0.0010 


Once I start something, I finish it 


0.0541 


0.0143 


3.7860 


0.0000 


Participated in SSS in first year 


0.0196 


0.0197 


0.9990 


0.3180 


Received non-SSS services in first year 


0.0042 


0.0132 


0.3210 


0.7480 


Received services after first year 


0.1279 


0.0201 


6.3780 


0.0000 


Missing data on student race 


0.0485 


0.0496 


0.9780 


0.3290 


Missing data on college housing 


0.0790 


0.1173 


0.6740 


0.5000 


Missing data on full-time status 


0.0966 


0.1388 


0.6960 


0.4860 


Missing data on high school GPA 


0.2809 


0.0409 


6.8670 


0.0000 


Missing data on SAT/ACT 


-0.0233 


0.0241 


-0.9630 


0.3360 


Missing data on developmental courses 


-0.1171 


0.0321 


-3.6470 


0.0000 


Missing data for hours on school activities 


0.0329 


0.0381 


0.8640 


0.3880 


Missing data on first generation 


-0.0459 


0.0346 


-1.3270 


0.1850 


Missing data on income 


0.0159 


0.0201 


0.7910 


0.4290 


Missing data on perceived ability 


0.0460 


0.0555 


0.8280 


0.4080 


Missing data on financial concerns 


-0.1006 


0.0486 


-2.0700 


0.0380 


Missing data on finishing tasks 


-0.0372 


0.0612 


-0.6080 


0.5430 


Missing data on services in later years 


0.0800 


0.0598 


1.3390 


0.1810 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-6. HLM analysis to predict associate’s degree completion (or higher) 



at any institution six years after freshman entry in 1991-92 using 
dichotomous measures of participation 



Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.5401 


0.0151 


35.8150 


0.0000 


Two-year institution 


-0.0412 


0.0349 


-1.1790 


0.2450 


Doctorate-granting institution 


0.0445 


0.0399 


1.1160 


0.2710 


Black student 


-0.0764 


0.0280 


-2.7240 


0.0070 


Hispanic student 


-0.0499 


0.0237 


-2.1020 


0.0350 


American Indian student 


-0.2518 


0.0551 


-4.5710 


0.0000 


Asian student 


0.0282 


0.0302 


0.9320 


0.3520 


Live in college housing or frat. /sorority 


0.0534 


0.0181 


2.9440 


0.0040 


Student was full-time 


0.0872 


0.0308 


2.8300 


0.0050 


High school GPA 


0.1184 


0.0157 


7.5480 


0.0000 


Took any developmental course 


-0.0960 


0.0146 


-6.5700 


0.0000 


Hours on school activities 


0.0014 


0.0006 


2.4830 


0.0130 


Attended college after first year 


0.3060 


0.0388 


7.8940 


0.0000 


No parental education beyond high school 


-0.0217 


0.0169 


-1.2810 


0.2000 


Family income greater than $20,000 


0.0190 


0.0174 


1.0890 


0.2770 


Above average academic ability 


0.0451 


0.0158 


2.8530 


0.0050 


Have major concerns about ed. finances 


-0.0357 


0.0142 


-2.5140 


0.0120 


Feel comfortable on this campus 


0.0480 


0.0206 


2.3320 


0.0200 


Participated in SSS in first year 


0.0005 


0.0197 


0.0230 


0.9820 


Received non-SSS services in first year 


0.0207 


0.0129 


1.6030 


0.1090 


Received services after first year 


0.1034 


0.0248 


4.1730 


0.0000 


Missing data on student race 


-0.0175 


0.0544 


-0.3210 


0.7480 


Missing data on college housing 


-0.0279 


0.1464 


-0.1900 


0.8490 


Missing data on full-time status 


0.1408 


0.1860 


0.7570 


0.4490 


Missing data on high school GPA 


0.3089 


0.0597 


5.1710 


0.0000 


Missing data on developmental courses 


-0.1356 


0.0384 


-3.5270 


0.0010 


Missing data for hours on school activities 


0.0219 


0.0400 


0.5490 


0.5830 


Missing data on first generation 


-0.0665 


0.0344 


-1.9310 


0.0530 


Missing data on income 


0.0147 


0.0190 


0.7730 


0.4400 


Missing data on perceived ability 


0.0250 


0.0721 


0.3460 


0.7290 


Missing data on financial concerns 


-0.0214 


0.0899 


-0.2380 


0.8120 


Missing data on comfort on campus 


0.0387 


0.0836 


0.4620 


0.6430 


Missing data on services in later years 


0.1240 


0.0632 


1.9630 


0.0490 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-7. HLM analysis to predict transfers from two-year to four-year 



institutions using dichotomous measures six years after 
freshman entry in 1991-92 of participation 



V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.2124 


0.0176 


12.0660 


0.0000 


Female student 


-0.0728 


0.0261 


-2.7880 


0.0060 


Live in college housing or frat./sorority 


0.3242 


0.0785 


4.1290 


0.0000 


Newly married 


-0.0960 


0.0368 


-2.6110 


0.0090 


High school GPA 


0.0483 


0.0181 


2.6680 


0.0080 


Took any developmental course 


-0.0789 


0.0259 


-3.0440 


0.0030 


Attended college after first year 


0.0355 


0.0358 


0.9920 


0.3220 


No parental education beyond high 
school 


-0.1133 


0.0265 


-4.2700 


0.0000 


Family income greater than $20,000 


0.0160 


0.0261 


0.6120 


0.5400 


Expect harder time than most 


-0.0531 


0.0281 


-1.8880 


0.0590 


Participated in SSS in first year 


-0.0043 


0.0284 


-0.1530 


0.8790 


Received non-SSS services in first year 


0.0410 


0.0269 


1.5270 


0.1270 


Received services after first year 


0.2118 


0.0311 


6.8200 


0.0000 


Missing data on college housing 


-0.0140 


0.0926 


-0.1510 


0.8800 


Missing data on high school GPA 


0.0714 


0.0696 


1.0250 


0.3060 


Missing data on developmental courses 


-0.1050 


0.0613 


-1.7120 


0.0860 


Missing data on first generation 


-0.0707 


0.0494 


-1.4310 


0.1520 


Missing data on income 


0.0122 


0.0359 


0.3390 


0.7350 


Missing data on expect harder time 


-0.0348 


0.0768 


-0.4530 


0.6500 


Missing data on services in later years 


-0.0149 


0.0708 


-0.2110 


0.8330 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-8. HLM analysis to predict cumulative GPAs six years after freshman entry 



in 1991-92 using continuous measures of participation and no measures 
of propensity to receive services 



Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


2.4089 


0.0248 


97.1910 


0.0000 


SSS services are home-based 


0.1215 


0.0667 


1.8220 


0.0750 


Two-year institution 


0.1369 


0.0623 


2.1980 


0.0340 


Doctorate -granting institution 


-0.0758 


0.0653 


-1.1600 


0.2530 


No SSS programs at institution 


0.0371 


0.0667 


0.5570 


0.5800 


SSS services are blended 


-0.0494 


0.0687 


-0.7190 


0.4760 


Age in 1991 


0.0272 


0.0020 


13.9180 


0.0000 


Black student 


-0.3281 


0.0307 


-10.7000 


0.0000 


Hispanic student 


-0.1416 


0.0393 


-3.6030 


0.0010 


American Indian student 


-0.2643 


0.0880 


-3.0050 


0.0030 


Asian student 


-0.0551 


0.0537 


-1.0250 


0.3060 


Female student 


0.1365 


0.0219 


6.2450 


0.0000 


High school GPA 


0.2883 


0.0194 


14.8380 


0.0000 


Percentile on SAT/ACT 


0.2148 


0.0547 


3.9230 


0.0000 


Took any developmental course 


-0.1453 


0.0235 


-6.1770 


0.0000 


Hours on school activities 


0.0034 


0.0007 


4.9680 


0.0000 


Attended college after first year 


0.7027 


0.0338 


20.7800 


0.0000 


No parental education beyond high school 


0.0243 


0.0230 


1.0560 


0.2910 


Family income greater than $20,000 


0.0249 


0.0239 


1.0430 


0.2980 


Above average academic ability 


0.1384 


0.0219 


6.3230 


0.0000 


Would attend free tutoring 


-0.0824 


0.0233 


-3.5320 


0.0010 


Feel comfortable on this campus 


0.0641 


0.0266 


2.4080 


0.0160 


Number of hours: instr. Courses 


0.0003 


0.0005 


0.5550 


0.5790 


Number of hours: profl. tutoring 


-0.0032 


0.0090 


-0.3570 


0.7210 


Number of hours: peer tutoring 


0.0059 


0.0013 


4.5090 


0.0000 


Number of hours: profl. counseling 


-0.0141 


0.0057 


-2.4650 


0.0140 


Number of hours: peer counseling 


-0.0331 


0.0217 


-1.5240 


0.1270 


Number of hours: labs 


0.0009 


0.0024 


0.3790 


0.7040 


Number of hours: workshops 


0.0106 


0.0062 


1.7180 


0.0850 


Number of hours: cultural events 


0.0086 


0.0094 


0.9060 


0.3650 


Number of hours: handicapped srvc. 


0.0006 


0.0034 


0.1670 


0.8670 


Services for physically disabled 


-0.0126 


0.0353 


-0.3560 


0.7210 


Services for limited-English ability 


0.0121 


0.0603 


0.2000 


0.8420 


Student orientation 


-0.0347 


0.0295 


-1.1750 


0.2400 


College reentrance counseling 


-0.0521 


0.0292 


-1.7860 


0.0740 


Classroom instr.— basic skills 


-0.0523 


0.0358 


-1.4610 


0.1440 


Classroom instr.— dvlpmntl. English 


0.0038 


0.0408 


0.0940 


0.9260 


Classroom instr.— dvlpmntl. math 


0.0086 


0.0361 


0.2370 


0.8130 


Cultural enrichment activities 


0.0118 


0.0053 


2.2050 


0.0270 


Referrals to agencies/resources 


0.0649 


0.0286 


2.2720 


0.0230 


# of counseling sessions in 1991-92 


-0.0042 


0.0027 


-1.5520 


0.1200 



Continues to next page. 
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# of counseling sessions in 1992-93 


0.0002 


0.0027 


0.0870 


0.9310 


# of counseling sessions in 1993-94 


0.0076 


0.0027 


2.8290 


0.0050 


# of tutoring sessions in 1991-92 


-0.0002 


0.0016 


-0.1480 


0.8830 


# of tutoring sessions in 1992-93 


-0.0005 


0.0017 


-0.3010 


0.7630 


# of tutoring sessions in 1993-94 


0.0013 


0.0019 


0.6960 


0.4860 


Some tutoring in years 4-6 


-0.0116 


0.0034 


-3.3910 


0.0010 


Some counseling in years 4-6 


0.0111 


0.0037 


2.9710 


0.0030 


Some services for disabled in years 4-6 


-0.0637 


0.0555 


-1.1490 


0.2510 


Field trip in years 4-6 


0.1254 


0.0265 


4.7270 


0.0000 


Contact with support services in years 4-6 


0.0164 


0.0031 


5.3740 


0.0000 


Missing data on student age 


0.5991 


0.0855 


7.0050 


0.0000 


Missing data on student race 


0.0296 


0.0856 


0.3460 


0.7290 


Missing data on high school GPA 


0.7650 


0.0791 


9.6740 


0.0000 


Missing data on SAT/ACT 


0.0356 


0.0348 


1.0230 


0.3070 


Missing data on developmental courses 


-0.1948 


0.0809 


-2.4080 


0.0160 


Missing data for hours on school activities 


0.1400 


0.0581 


2.4080 


0.0160 


Missing data on first generation 


-0.0917 


0.0516 


-1.7760 


0.0750 


Missing data on income 


-0.0212 


0.0355 


-0.5990 


0.5490 


Missing data on perceived ability 


0.1268 


0.1051 


1.2060 


0.2280 


Missing data on attending free tutoring 


0.0243 


0.1935 


0.1250 


0.9010 


Missing data on comfort on campus 


-0.2178 


0.1928 


-1.1290 


0.2590 


Missing data on services in later years 


0.2291 


0.0686 


3.3380 


0.0010 


Missing data on services for limited-English 


-0.1189 


0.4893 


-0.2430 


0.8080 


Missing data on student orientation 


-0.1051 


0.1237 


-0.8490 


0.3960 


Missing data on re-entrance counseling 


0.1733 


0.0712 


2.4350 


0.0150 


Missing data on classroom instr.— basic skills 


-0.1131 


0.1043 


-1.0850 


0.2790 


Missing data on cultural enrichment activities 


-0.2931 


0.2373 


-1.2350 


0.2170 


Missing data on referrals to agencies/resources 


0.5182 


0.5420 


0.9560 


0.3390 


Missing data on tutoring in 1991-92 


-0.0611 


0.0753 


-0.8110 


0.4180 


Missing data on tutoring in years 4-6 


-0.0929 


0.1311 


-0.7090 


0.4780 


Missing data on counseling in years 4-6 


-0.0660 


0.0882 


-0.7490 


0.4540 


Missing data on services for disabled in years 4-6 


-0.0711 


0.1298 


-0.5480 


0.5830 


Missing data on field trip in years 4-6 


0.4276 


0.1805 


2.3700 


0.0180 


Missing data on contact with support services in yrs. 
4-6 


-0.2436 


0.1759 


-1.3850 


0.1660 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-9. HLM analysis to predict the number of credits earned six years after 
freshman entry in 1991-92 using continuous measures of participation 



and no measures of propensity to receive services 


V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 



Intercept 

SSS services are home-based 



Two-year institution 
Doctorate -granting institution 
No SSS programs at institution 
SSS services are blended 
Black student 
Hispanic student 
American Indian student 
Asian student 
Female student 
Student was full-time 
High school GPA 
Percentile on SAT/ACT 
Took any developmental course 
Hours on school activities 
Attended college after first year 
No parental education beyond high school 
Family income greater than $20,000 
Have major concerns about ed. finances 
Once I start something, I finish it 
Feel comfortable on this campus 
Number of hours: instr. courses 
Number of hours: profl. tutoring 
Number of hours: peer tutoring 
Number of hours: profl. counseling 
Number of hours: peer counseling 
Number of hours: labs 
Number of hours: workshops 
Number of hours: cultural events 
Number of hours: handicapped srvc. 
Services for physically disabled 
Services for limited-English ability 
Student orientation 
College reentrance counseling 
Classroom instr.— basic skills 
Classroom instr.— dvlpmntl. English 
Classroom instr.— dvlpmntl. math 
Cultural enrichment activities 
Referrals to agencies/resources 
# of counseling sessions in 1991-92 



76.1473 


1.7824 


42.7210 


0.0000 


4.5830 


4.8039 


0.9540 


0.3460 


-14.5154 


4.4250 


-3.2800 


0.0020 


4.1163 


4.7266 


0.8710 


0.3890 


3.1919 


4.7665 


0.6700 


0.5070 


-2.5910 


4.9978 


-0.5180 


0.6060 


-6.3235 


1.7780 


-3.5560 


0.0010 


-1.3367 


2.2859 


-0.5850 


0.5580 


-18.8912 


5.1443 


-3.6720 


0.0000 


6.4593 


3.0902 


2.0900 


0.0360 


3.9224 


1.2552 


3.1250 


0.0020 


13.7099 


2.2520 


6.0880 


0.0000 


10.2556 


1.1129 


9.2150 


0.0000 


11.3196 


3.1250 


3.6220 


0.0010 


-6.0028 


1.3548 


-4.4310 


0.0000 


0.1716 


0.0395 


4.3460 


0.0000 


43.7484 


1.9238 


22.7410 


0.0000 


-0.7331 


1.3206 


-0.5550 


0.5780 


-1.8817 


1.3812 


-1.3620 


0.1730 


-4.1052 


1.2510 


-3.2820 


0.0010 


3.6691 


1.4058 


2.6100 


0.0090 


6.8203 


1.5466 


4.4100 


0.0000 


0.0514 


0.0278 


1.8520 


0.0640 


-0.3803 


0.5231 


-0.7270 


0.4670 


0.3910 


0.0758 


5.1580 


0.0000 


-0.8865 


0.3333 


-2.6600 


0.0080 


-0.6247 


1.2633 


-0.4940 


0.6210 


-0.0248 


0.1371 


-0.1810 


0.8570 


0.0276 


0.3594 


0.0770 


0.9390 


1.4762 


0.5487 


2.6910 


0.0080 


0.1528 


0.1954 


0.7820 


0.4340 


1.5046 


2.0381 


0.7380 


0.4600 


0.3152 


3.4657 


0.0910 


0.9280 


-0.3259 


1.7005 


-0.1920 


0.8480 


-4.2336 


1.6798 


-2.5200 


0.0120 


-5.0660 


2.0673 


-2.4500 


0.0140 


1.2470 


2.3485 


0.5310 


0.5950 


-1.6766 


2.0798 


-0.8060 


0.4200 


0.6065 


0.3078 


1.9700 


0.0480 


-1.7362 


1.6520 


-1.0510 


0.2940 


-0.1195 


0.1567 


-0.7620 


0.4460 



Continues to next page. 
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# of counseling sessions in 1992-93 


0.0498 


0.1554 


0.3200 


0.7480 


# of counseling sessions in 1993-94 


0.4968 


0.1562 


3.1810 


0.0020 


# of tutoring sessions in 1991-92 


-0.1099 


0.0927 


-1.1850 


0.2360 


# of tutoring sessions in 1992-93 


0.4678 


0.0980 


4.7740 


0.0000 


# of tutoring sessions in 1993-94 


0.3040 


0.1106 


2.7480 


0.0060 


Some tutoring in years 4-6 


0.0091 


0.1977 


0.0460 


0.9640 


Some counseling in years 4-6 


1.4312 


0.2165 


6.6100 


0.0000 


Some services for disabled in years 4-6 


-12.2753 


3.2296 


-3.8010 


0.0000 


Field trip in years 4-6 


12.9644 


1.5348 


8.4470 


0.0000 


Contact with support services in years 4-6 


2.1192 


0.1768 


11.9850 


0.0000 


Missing data on student race 


-3.9126 


4.9972 


-0.7830 


0.4340 


Missing data on full-time status 


20.6079 


7.1663 


2.8760 


0.0040 


Missing data on high school GPA 


26.7383 


4.5542 


5.8710 


0.0000 


Missing data on SAT/ACT 


0.2429 


1.9648 


0.1240 


0.9020 


Missing data on developmental courses 


-2.6660 


4.6790 


-0.5700 


0.5680 


Missing data for hours on school activities 


5.3797 


3.5652 


1.5090 


0.1310 


Missing data on first generation 


-2.2715 


3.0565 


-0.7430 


0.4570 


Missing data on income 


-1.8068 


2.0460 


-0.8830 


0.3770 


Missing data on perceived ability 


1.6111 


6.1057 


0.2640 


0.7920 


Missing data on financial concerns 


-8.8075 


5.6964 


-1.5460 


0.1220 


Missing data on finishing tasks 


9.3706 


6.0911 


1.5380 


0.1240 


Missing data on comfort on campus 


-5.2128 


7.5539 


-0.6900 


0.4900 


Missing data on B32 


-1.6215 


3.9603 


-0.4090 


0.6820 


Missing data on services for limited-English 


-20.8979 


28.1960 


-0.7410 


0.4590 


Missing data on student orientation 


0.8228 


7.1110 


0.1160 


0.9080 


Missing data on reentrance counseling 


-1.0399 


4.1092 


-0.2530 


0.8000 


Missing data on classroom instr.— basic skills 


7.7590 


5.8813 


1.3190 


0.1870 


Missing data on cultural enrichment activities 


-29.4697 


13.6924 


-2.1520 


0.0310 


Missing data on referrals to agencies/resources 


45.3459 


31.2358 


1.4520 


0.1470 


Missing data on tutoring in 1991-92 


-3.0130 


4.3513 


-0.6920 


0.4880 


Missing data on tutoring in years 4-6 


-0.0991 


7.6381 


-0.0130 


0.9900 


Missing data on counseling in years 4-6 


-5.3820 


5.0868 


-1.0580 


0.2910 


Missing data on services for disabled in years 4-6 


-8.6346 


7.5275 


-1.1470 


0.2520 


Missing data on field trip in years 4-6 


7.6529 


10.6411 


0.7190 


0.4720 


Missing data on contact with support services in yrs. 
4-6 


6.0809 


10.3418 


0.5880 


0.5560 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 



87 




Exhibit A- 10. HLM analysis to predict retention or baccalaureate degree completion 



at the same institution six years after freshman entry in 1991-92 using 
continuous measures of participation and no measures of propensity 
to receive services 



V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.3184 


0.0109 


29.2890 


0.0000 


SSS services are home-based 


0.0545 


0.0291 


1.8760 


0.0670 


Two-year institution 


-0.2288 


0.0279 


-8.2000 


0.0000 


Doctorate -granting institution 


0.0433 


0.0284 


1.5240 


0.1350 


No SSS programs at institution 


0.0561 


0.0297 


1.8900 


0.0650 


SSS services are blended 


0.0692 


0.0292 


2.3730 


0.0230 


Female student 


0.0295 


0.0128 


2.3050 


0.0210 


High school GPA 


0.0620 


0.0111 


5.6070 


0.0000 


Percentile on SAT/ACT 


0.1309 


0.0314 


4.1620 


0.0000 


Took any developmental course 


-0.0511 


0.0139 


-3.6810 


0.0000 


Attended college after first year 


0.1635 


0.0197 


8.3030 


0.0000 


No parental education beyond high school 


0.0029 


0.0135 


0.2160 


0.8290 


Family income greater than $20,000 


-0.0198 


0.0139 


-1.4250 


0.1540 


Feel comfortable on this campus 


0.0630 


0.0156 


4.0250 


0.0000 


Number of hours: instr. courses 


0.0003 


0.0003 


1.0160 


0.3100 


Number of hours: profl. tutoring 


-0.0021 


0.0053 


-0.3960 


0.6920 


Number of hours: peer tutoring 


0.0026 


0.0008 


3.3280 


0.0010 


Number of hours: profl. counseling 


-0.0109 


0.0034 


-3.2410 


0.0020 


Number of hours: peer counseling 


-0.0009 


0.0127 


-0.0730 


0.9420 


Number of hours: labs 


0.0003 


0.0014 


0.2480 


0.8050 


Number of hours: workshops 


0.0051 


0.0036 


1.4090 


0.1590 


Number of hours: cultural events 


0.0029 


0.0055 


0.5180 


0.6040 


Number of hours: handicapped srvc. 


0.0058 


0.0020 


2.8930 


0.0040 


Services for physically disabled 


0.0343 


0.0209 


1.6410 


0.1000 


Services for limited-English ability 


0.0266 


0.0353 


0.7550 


0.4500 


Student orientation 


0.0127 


0.0173 


0.7330 


0.4640 


College reentrance counseling 


-0.0184 


0.0172 


-1.0680 


0.2860 


Classroom instr.— basic skills 


-0.0190 


0.0213 


-0.8940 


0.3720 


Classroom instr.— dvlpmntl. English 


0.0032 


0.0241 


0.1320 


0.8960 


Classroom instr.— dvlpmntl. math 


-0.0048 


0.0214 


-0.2250 


0.8220 


Cultural enrichment activities 


-0.0022 


0.0032 


-0.7050 


0.4810 


Referrals to agencies/resources 


-0.0198 


0.0169 


-1.1750 


0.2400 


# of counseling sessions in 1991-92 


-0.0022 


0.0016 


-1.4050 


0.1600 


# of counseling sessions in 1992-93 


-0.0005 


0.0016 


-0.2880 


0.7730 


# of counseling sessions in 1993-94 


0.0036 


0.0016 


2.2370 


0.0250 


# of tutoring sessions in 1991-92 


-0.0010 


0.0009 


-1.0520 


0.2930 


# of tutoring sessions in 1992-93 


0.0017 


0.0010 


1.7350 


0.0820 


# of tutoring sessions in 1993-94 


0.0025 


0.0011 


2.1610 


0.0300 


Some tutoring in years 4-6 


0.0015 


0.0020 


0.7350 


0.4620 


Some counseling in years 4-6 


0.0113 


0.0022 


5.0640 


0.0000 
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Some services for disabled in years 4-6 


-0.0735 


0.0327 


-2.2450 


0.0250 


Field trip in years 4-6 


0.0740 


0.0158 


4.6990 


0.0000 


Contact with support services in years 4-6 


0.0177 


0.0018 


9.7580 


0.0000 


Missing data on high school GPA 


0.1997 


0.0449 


4.4490 


0.0000 


Missing data on SAT/ACT 


0.0361 


0.0197 


1.8270 


0.0670 


Missing data on developmental courses 


-0.1330 


0.0479 


-2.7750 


0.0060 


Missing data for hours on school activities 


-0.0070 


0.0318 


-0.2200 


0.8260 


Missing data on first generation 


0.0206 


0.0297 


0.6930 


0.4880 


Missing data on income 


-0.0148 


0.0207 


-0.7160 


0.4740 


Missing data on comfort on campus 


-0.0059 


0.0514 


-0.1150 


0.9090 


Missing data on services in later years 


0.0615 


0.0406 


1.5150 


0.1290 


Missing data on services for limited-English 


-0.4768 


0.2914 


-1.6370 


0.1010 


Missing data on student orientation 


0.0104 


0.0714 


0.1450 


0.8850 


Missing data on reentrance counseling 


0.0076 


0.0416 


0.1820 


0.8560 


Missing data on classroom instr.— basic skills 


0.0391 


0.0601 


0.6510 


0.5150 


Missing data on cultural enrichment activities 


0.2479 


0.1411 


1.7570 


0.0790 


Missing data on referrals to agencies/resources 


0.1910 


0.3226 


0.5920 


0.5540 


Missing data on tutoring in 1991-92 


0.0173 


0.0447 


0.3870 


0.6990 


Missing data on tutoring in years 4-6 


-0.0060 


0.0766 


-0.0780 


0.9380 


Missing data on counseling in years 4-6 


-0.0510 


0.0523 


-0.9760 


0.3300 


Missing data on services for disabled in years 4-6 


-0.0059 


0.0772 


-0.0770 


0.9390 


Missing data on field trip in years 4-6 


0.0540 


0.1070 


0.5050 


0.6130 


Missing data on contact with support services in yrs. 
4-6 


0.0102 


0.1045 


0.0970 


0.9230 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-ll. HLM analysis to predict retention or baccalaureate degree completion 



at any institution six years after freshman entry in 1991-92 using 
continuous measures of participation and no measures of propensity to 
receive services 



V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.6167 


0.0101 


61.2400 


0.0000 


SSS services are home-based 


0.0766 


0.0269 


2.8420 


0.0070 


Two-year institution 


-0.1518 


0.0262 


-5.7910 


0.0000 


Doctorate -granting institution 


0.0323 


0.0259 


1.2470 


0.2200 


No SSS programs at institution 


0.0701 


0.0276 


2.5370 


0.0150 


SSS services are blended 


0.0647 


0.0266 


2.4300 


0.0200 


Female student 


0.0463 


0.0131 


3.5450 


0.0010 


High school GPA 


0.0459 


0.0114 


4.0430 


0.0000 


Percentile on SAT/ACT 


0.0827 


0.0317 


2.6110 


0.0090 


Took any developmental course 


-0.0526 


0.0138 


-3.8160 


0.0000 


Hours on school activities 


0.0005 


0.0004 


1.2350 


0.2170 


Attended college after first year 


0.3491 


0.0234 


14.9450 


0.0000 


No parental education beyond high school 


-0.0468 


0.0136 


-3.4370 


0.0010 


Family income greater than $20,000 


0.0008 


0.0141 


0.0590 


0.9530 


Above average academic ability 


0.0114 


0.0130 


0.8740 


0.3820 


Number of hours: instr. courses 


-0.0002 


0.0003 


-0.8800 


0.3790 


Number of hours: profl. tutoring 


-0.0026 


0.0057 


-0.4580 


0.6470 


Number of hours: peer tutoring 


0.0029 


0.0008 


3.7890 


0.0000 


Number of hours: profl. counseling 


-0.0015 


0.0034 


-0.4320 


0.6650 


Number of hours: peer counseling 


-0.0083 


0.0129 


-0.6460 


0.5180 


Number of hours: labs 


-0.0006 


0.0014 


-0.4640 


0.6420 


Number of hours: workshops 


-0.0003 


0.0036 


-0.0860 


0.9320 


Number of hours: cultural events 


-0.0014 


0.0054 


-0.2620 


0.7930 


Number of hours: handicapped srvc. 


0.0017 


0.0019 


0.9320 


0.3520 


Services for physically disabled 


-0.0141 


0.0238 


-0.5910 


0.5540 


Services for limited-English ability 


0.0647 


0.0370 


1.7460 


0.0800 


Student orientation 


0.0170 


0.0177 


0.9640 


0.3350 


College reentrance counseling 


0.0052 


0.0191 


0.2730 


0.7850 


Classroom instr.— basic skills 


-0.0068 


0.0216 


-0.3150 


0.7530 


Classroom instr.— dvlpmntl. English 


-0.0237 


0.0243 


-0.9770 


0.3290 


Classroom instr.— dvlpmntl. math 


-0.0137 


0.0217 


-0.6330 


0.5270 


Cultural enrichment activities 


0.0030 


0.0035 


0.8450 


0.3980 


Referrals to agencies/resources 


-0.0280 


0.0170 


-1.6440 


0.1000 


# of counseling sessions in 1991-92 


-0.0032 


0.0016 


-1.9630 


0.0490 


# of counseling sessions in 1992-93 


-0.0015 


0.0015 


-1.0020 


0.3170 


# of counseling sessions in 1993-94 


0.0033 


0.0015 


2.1780 


0.0290 


# of tutoring sessions in 1991-92 


-0.0011 


0.0009 


-1.1760 


0.2400 


# of tutoring sessions in 1992-93 


0.0005 


0.0010 


0.5460 


0.5850 


# of tutoring sessions in 1993-94 


0.0047 


0.0011 


4.1940 


0.0000 


Some tutoring in years 4-6 


-0.0016 


0.0019 


-0.8310 


0.4060 
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Some counseling in years 4-6 


0.0209 


0.0021 


10.0440 


0.0000 


Some services for disabled in years 4-6 


-0.0308 


0.0306 


-1.0060 


0.3150 


Field trip in years 4-6 


0.1337 


0.0147 


9.0900 


0.0000 


Contact with support services in years 4-6 


0.0193 


0.0017 


11.3650 


0.0000 


Missing data on full-time status 


0.0064 


0.0662 


0.0960 


0.9240 


Missing data on high school GPA 


0.1174 


0.0456 


2.5760 


0.0100 


Missing data on SAT/ACT 


-0.0267 


0.0200 


-1.3360 


0.1820 


Missing data on developmental courses 


0.0302 


0.0540 


0.5590 


0.5760 


Missing data for hours on school activities 


0.0017 


0.0384 


0.0450 


0.9640 


Missing data on first generation 


-0.0338 


0.0334 


-1.0110 


0.3130 


Missing data on income 


-0.0159 


0.0214 


-0.7430 


0.4580 


Missing data on perceived ability 


0.0640 


0.0563 


1.1370 


0.2560 


Missing data on B32 


-0.0522 


0.0468 


-1.1150 


0.2650 


Missing data on services for limited-English 


-0.1851 


0.2716 


-0.6810 


0.4950 


Missing data on student orientation 


0.0444 


0.0743 


0.5970 


0.5500 


Missing data on reentrance counseling 


0.0125 


0.0406 


0.3070 


0.7580 


Missing data on classroom instr.— basic skills 


-0.0182 


0.0603 


-0.3020 


0.7630 


Missing data on cultural enrichment activities 


-0.2675 


0.1320 


-2.0260 


0.0420 


Missing data on referrals to agencies/resources 


0.4141 


0.3010 


1.3760 


0.1690 


Missing data on tutoring in 1991-92 


-0.0901 


0.0572 


-1.5760 


0.1150 


Missing data on tutoring in years 4-6 


0.1907 


0.0720 


2.6510 


0.0080 


Missing data on counseling in years 4-6 


-0.0330 


0.0487 


-0.6760 


0.4990 


Missing data on services for disabled in years 4-6 


-0.1130 


0.0719 


-1.5720 


0.1160 


Missing data on field trip in years 4-6 

Missing data on contact with support services in yrs. 


0.2344 


0.0997 


2.3510 


0.0190 


4-6 


0.1208 


0.0975 


1.2400 


0.2150 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 12. HLM analysis to predict baccalaureate degree completion (or higher) at any 
institution six years after freshman entry in 1991-92 using continuous 



measures of participation and no measures ol 


f propensity to receive services 


V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 



Intercept 

SSS services are home-based 



Two-year institution 
Doctorate -granting institution 
No SSS programs at institution 
SSS services are blended 
Black student 
Hispanic student 
American Indian student 
Asian student 
Student was female 

Live in college housing or frat./sorority 
High school GPA 
Percentile on SAT/ACT 
Took any developmental course 
Hours on school activities 
Attended college after first year 
No parental education beyond high school 
Family income greater than $20,000 
Have major concerns about ed. finances 
Once I start something, I finish it 
Number of hours: instr. courses 
Number of hours: profl. tutoring 
Number of hours: peer tutoring 
Number of hours: profl. counseling 
Number of hours: peer counseling 
Number of hours: labs 
Number of hours: workshops 
Number of hours: cultural events 
Number of hours: handicapped srvc. 
Services for physically disabled 
Services for limited-English ability 
Student orientation 
College reentrance counseling 
Classroom instr.— basic skills 
Classroom instr.— dvlpmntl. English 
Classroom instr.— dvlpmntl. math 
Cultural enrichment activities 
Referrals to agencies/resources 

# of counseling sessions in 1991-92 

# of counseling sessions in 1992-93 



0.3973 


0.0137 


28.9700 


0.0000 


0.0520 


0.0370 


1.4060 


0.1670 


-0.1663 


0.0351 


-4.7320 


0.0000 


0.0633 


0.0358 


1.7660 


0.0840 


0.0413 


0.0370 


1.1160 


0.2710 


0.0251 


0.0377 


0.6650 


0.5090 


-0.0411 


0.0191 


-2.1500 


0.0310 


-0.0047 


0.0241 


-0.1960 


0.8450 


-0.1626 


0.0549 


-2.9600 


0.0040 


0.0989 


0.0337 


2.9330 


0.0040 


0.0249 


0.0136 


1.8260 


0.0670 


0.0580 


0.0168 


3.4560 


0.0010 


0.0960 


0.0119 


8.0580 


0.0000 


0.1474 


0.0334 


4.4130 


0.0000 


-0.0875 


0.0143 


-6.1000 


0.0000 


0.0011 


0.0004 


2.4640 


0.0140 


0.1448 


0.0242 


5.9870 


0.0000 


-0.0229 


0.0143 


-1.6020 


0.1090 


0.0161 


0.0149 


1.0800 


0.2810 


-0.0451 


0.0136 


-3.3200 


0.0010 


0.0424 


0.0151 


2.8080 


0.0050 


0.0002 


0.0003 


0.6140 


0.5390 


0.0008 


0.0060 


0.1390 


0.8900 


0.0023 


0.0008 


2.8690 


0.0050 


-0.0089 


0.0035 


-2.4980 


0.0130 


-0.0011 


0.0139 


-0.0790 


0.9370 


-0.0004 


0.0015 


-0.2420 


0.8090 


-0.0012 


0.0038 


-0.3210 


0.7480 


0.0052 


0.0057 


0.9130 


0.3620 


-0.0004 


0.0020 


-0.2050 


0.8380 


0.0179 


0.0247 


0.7260 


0.4680 


0.0098 


0.0391 


0.2510 


0.8020 


0.0083 


0.0186 


0.4490 


0.6530 


-0.0192 


0.0198 


-0.9700 


0.3330 


-0.0048 


0.0225 


-0.2150 


0.8300 


-0.0010 


0.0254 


-0.0410 


0.9680 


-0.0150 


0.0227 


-0.6610 


0.5090 


-0.0016 


0.0036 


-0.4530 


0.6500 


-0.0035 


0.0179 


-0.1950 


0.8460 


-0.0033 


0.0017 


-1.9530 


0.0500 


0.0021 


0.0016 


1.3360 


0.1820 
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# of counseling sessions in 1993-94 


0.0041 


0.0016 


2.5750 


0.0100 


# of tutoring sessions in 1991-92 


0.0005 


0.0010 


0.4700 


0.6380 


# of tutoring sessions in 1992-93 


0.0011 


0.0010 


1.0490 


0.2940 


# of tutoring sessions in 1993-94 


0.0003 


0.0012 


0.2270 


0.8210 


Some tutoring in years 4-6 


-0.0042 


0.0020 


-2.1470 


0.0320 


Some counseling in years 4-6 


0.0118 


0.0022 


5.4970 


0.0000 


Some services for disabled in years 4-6 


-0.1077 


0.0316 


-3.4100 


0.0010 


Field trip in years 4-6 


0.1345 


0.0152 


8.8600 


0.0000 


Contact with support services in years 4-6 


0.0163 


0.0018 


9.2790 


0.0000 


Missing data on student race 


0.0205 


0.0531 


0.3860 


0.6990 


Missing data on college housing 


0.1069 


0.0781 


1.3690 


0.1710 


Missing data on high school GPA 


0.2676 


0.0481 


5.5660 


0.0000 


Missing data on SAT/ACT 


-0.0050 


0.0211 


-0.2390 


0.8110 


Missing data on developmental courses 


-0.0298 


0.0575 


-0.5180 


0.6040 


Missing data for hours on school activities 


0.0468 


0.0402 


1.1650 


0.2450 


Missing data on first generation 


-0.0530 


0.0351 


-1.5080 


0.1310 


Missing data on income 


0.0216 


0.0225 


0.9630 


0.3360 


Missing data on financial concerns 


-0.0428 


0.0645 


-0.6630 


0.5070 


Missing data on finishing tasks 


-0.0294 


0.0557 


-0.5280 


0.5970 


Missing data on B32 


0.0144 


0.0503 


0.2850 


0.7750 


Missing data on services for limited-English 


0.5156 


0.2800 


1.8410 


0.0650 


Missing data on student orientation 


0.0193 


0.0771 


0.2510 


0.8020 


Missing data on reentrance counseling 


0.0088 


0.0422 


0.2090 


0.8340 


Missing data on classroom instr.— basic skills 


0.0192 


0.0622 


0.3090 


0.7580 


Missing data on cultural enrichment activities 


-0.1149 


0.1364 


-0.8420 


0.4000 


Missing data on referrals to agencies/resources 


-0.4777 


0.3103 


-1.5400 


0.1230 


Missing data on tutoring in 1991-92 


-0.0837 


0.0617 


-1.3560 


0.1750 


Missing data on tutoring in years 4-6 


0.1515 


0.0748 


2.0260 


0.0420 


Missing data on counseling in years 4-6 


0.0394 


0.0503 


0.7830 


0.4330 


Missing data on services for disabled in years 4-6 


-0.1299 


0.0742 


-1.7500 


0.0800 


Missing data on field trip in years 4-6 


0.3293 


0.1033 


3.1890 


0.0020 


Missing data on contact with support services in yrs. 
4-6 


0.0853 


0.1007 


0.8470 


0.3970 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 13. HLM analysis to predict associate’s degree completion (or higher) at any 
institution six years after freshman entry in 1991-92 using continuous 



measures of participation and no measures ol 


f propensity to receive services 


V ariable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 



Intercept 

SSS services are home-based 



Two-year institution 
Doctorate -granting institution 
No SSS programs at institution 
SSS services are blended 
Black student 
Hispanic student 
American Indian student 
Asian student 
Student was female 
Student was full-time 
High school GPA 
Took any developmental course 
Attended college after first year 
No parental education beyond high school 
Family income greater than $20,000 
Above average academic ability 
Have major concerns about ed. finances 
Number of hours: instr. courses 
Number of hours: profl. tutoring 
Number of hours: peer tutoring 
Number of hours: profl. counseling 
Number of hours: peer counseling 
Number of hours: labs 
Number of hours: workshops 
Number of hours: cultural events 
Number of hours: handicapped srv. 
Services for physically disabled 
Services for limited-English ability 
Student orientation 
College reentrance counseling 
Classroom instr.— basic skills 
Classroom instr.— dvlpmntl. English 
Classroom instr.— dvlpmntl. math 
Cultural enrichment activities 
Referrals to agencies/resources 

# of counseling sessions in 1991-92 

# of counseling sessions in 1992-93 

# of counseling sessions in 1993-94 

# of tutoring sessions in 1991-92 

Continues to next page. 



0.5381 


0.0148 


36.3290 


0.0000 


0.0258 


0.0398 


0.6490 


0.5200 


0.0005 


0.0372 


0.0120 


0.9900 


0.0523 


0.0388 


1.3490 


0.1850 


0.0099 


0.0399 


0.2480 


0.8050 


-0.0124 


0.0408 


-0.3050 


0.7620 


-0.0929 


0.0202 


-4.5970 


0.0000 


-0.0658 


0.0259 


-2.5350 


0.0120 


-0.2512 


0.0593 


-4.2380 


0.0000 


0.0385 


0.0363 


1.0610 


0.2890 


0.0536 


0.0148 


3.6190 


0.0010 


0.0939 


0.0265 


3.5400 


0.0010 


0.1035 


0.0129 


8.0380 


0.0000 


-0.0891 


0.0153 


-5.8220 


0.0000 


0.2742 


0.0262 


10.4650 


0.0000 


-0.0221 


0.0154 


-1.4290 


0.1530 


0.0160 


0.0161 


0.9940 


0.3210 


0.0345 


0.0146 


2.3670 


0.0180 


-0.0391 


0.0147 


-2.6620 


0.0080 


0.0002 


0.0003 


0.4710 


0.6370 


-0.0066 


0.0064 


-1.0400 


0.2990 


0.0021 


0.0009 


2.4340 


0.0150 


-0.0088 


0.0039 


-2.2820 


0.0220 


0.0008 


0.0150 


0.0510 


0.9600 


-0.0046 


0.0016 


-2.8440 


0.0050 


0.0062 


0.0041 


1.5050 


0.1320 


0.0010 


0.0062 


0.1550 


0.8770 


0.0014 


0.0021 


0.6790 


0.4970 


0.0115 


0.0261 


0.4410 


0.6590 


0.0065 


0.0421 


0.1540 


0.8780 


0.0179 


0.0201 


0.8930 


0.3720 


-0.0084 


0.0214 


-0.3900 


0.6960 


-0.0145 


0.0244 


-0.5960 


0.5510 


-0.0100 


0.0275 


-0.3620 


0.7170 


-0.0221 


0.0246 


-0.8980 


0.3690 


-0.0016 


0.0039 


-0.4030 


0.6860 


0.0153 


0.0193 


0.7940 


0.4270 


-0.0018 


0.0018 


-0.9660 


0.3340 


0.0007 


0.0017 


0.4050 


0.6850 


0.0052 


0.0017 


3.0190 


0.0030 


0.0004 


0.0011 


0.3640 


0.7160 
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# of tutoring sessions in 1992-93 


0.0013 


0.0011 


1.1350 


0.2570 


# of tutoring sessions in 1993-94 


0.0002 


0.0013 


0.1790 


0.8580 


Some tutoring in years 4-6 


-0.0052 


0.0021 


-2.4370 


0.0150 


Some counseling in years 4-6 


0.0088 


0.0023 


3.7350 


0.0000 


Some services for disabled in years 4-6 


-0.1053 


0.0344 


-3.0620 


0.0030 


Field trip in years 4-6 


0.1188 


0.0165 


7.1870 


0.0000 


Contact with support services in years 4-6 


0.0179 


0.0019 


9.4140 


0.0000 


Missing data on student race 


-0.0624 


0.0577 


-1.0810 


0.2800 


Missing data on full-time status 


0.1267 


0.0800 


1.5830 


0.1130 


Missing data on high school GPA 


0.2653 


0.0519 


5.1100 


0.0000 


Missing data on developmental courses 


-0.1008 


0.0625 


-1.6130 


0.1060 


Missing data on first generation 


-0.0720 


0.0380 


-1.8930 


0.0580 


Missing data on income 


0.0111 


0.0243 


0.4570 


0.6470 


Missing data on perceived ability 


0.0441 


0.0674 


0.6550 


0.5120 


Missing data on financial concerns 


0.0093 


0.0700 


0.1320 


0.8950 


Missing data on B32 


0.0174 


0.0538 


0.3240 


0.7460 


Missing data on services for limited-English 


0.1449 


0.3052 


0.4750 


0.6350 


Missing data on student orientation 


0.0142 


0.0834 


0.1710 


0.8650 


Missing data on reentrance counseling 


-0.0060 


0.0458 


-0.1320 


0.8950 


Missing data on classroom instr.— basic skills 


0.0349 


0.0677 


0.5160 


0.6060 


Missing data on cultural enrichment activities 


-0.1672 


0.1486 


-1.1260 


0.2610 


Missing data on referrals to agencies/resources 


-0.0284 


0.3382 


-0.0840 


0.9340 


Missing data on tutoring in 1991-92 


-0.0116 


0.0672 


-0.1730 


0.8630 


Missing data on tutoring in years 4-6 


0.1392 


0.0818 


1.7020 


0.0880 


Missing data on counseling in years 4-6 


0.0586 


0.0548 


1.0690 


0.2850 


Missing data on services for disabled in years 4-6 


0.0078 


0.0809 


0.0970 


0.9230 


Missing data on field trip in years 4-6 

Missing data on contact with support services in yrs. 


0.1999 


0.1123 


1.7800 


0.0750 


4-6 


0.0568 


0.1097 


0.5180 


0.6040 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 14. HLM analysis to predict transfers from two-year to four- year institutions six 



years after freshman entry in 1991-92 using continuous measures of 
participation and no measures of propensity to receive services 


Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.2155 


0.0175 


12.3210 


0.0000 


SSS services are home-based 


0.0473 


0.0603 


0.7830 


0.4490 


No SSS programs at institution 


0.0868 


0.0438 


1.9820 


0.0700 


SSS services are blended 


0.0142 


0.0491 


0.2890 


0.7770 


Female 


-0.0837 


0.0245 


-3.4150 


0.0010 


High school GPA 


0.0531 


0.0178 


2.9920 


0.0030 


Took any developmental course 


-0.0555 


0.0253 


-2.1990 


0.0280 


Attended college after first year 


0.0923 


0.0294 


3.1430 


0.0020 


No parental education beyond high school 


-0.1112 


0.0255 


-4.3570 


0.0000 


Family income greater than $20,000 


-0.0106 


0.0251 


-0.4220 


0.6720 


Once I start something, I finish it 


-0.0337 


0.0261 


-1.2910 


0.1970 


Number of hours: instr. courses 


-0.0003 


0.0008 


-0.3800 


0.7040 


Number of hours: profl. tutoring 


-0.0159 


0.0091 


-1.7430 


0.0810 


Number of hours: peer tutoring 


0.0021 


0.0017 


1.2020 


0.2300 


Number of hours: profl. counseling 


0.0097 


0.0060 


1.6040 


0.1080 


Number of hours: peer counseling 


0.0572 


0.0565 


1.0130 


0.3120 


Number of hours: labs 


0.0008 


0.0015 


0.5440 


0.5860 


Number of hours: workshops 


-0.0196 


0.0084 


-2.3370 


0.0200 


Number of hours: cultural events 


-0.0388 


0.0170 


-2.2870 


0.0220 


Number of hours: handicapped srvc. 


-0.0060 


0.0031 


-1.9520 


0.0510 


Services for physically disabled 


0.0202 


0.0328 


0.6160 


0.5370 


Services for limited-English ability 


-0.0480 


0.0564 


-0.8520 


0.3940 


Student orientation 


0.0096 


0.0256 


0.3730 


0.7080 


College reentrance counseling 


0.0542 


0.0261 


2.0750 


0.0380 


Classroom instr.— basic skills 


-0.0690 


0.0392 


-1.7590 


0.0780 


Classroom instr.— dvlpmntl. English 


0.0332 


0.0417 


0.7970 


0.4260 


Classroom instr.— dvlpmntl. math 


-0.0769 


0.0383 


-2.0080 


0.0440 


Cultural enrichment activities 


-0.0134 


0.0181 


-0.7420 


0.4580 


Referrals to agencies/resources 


-0.0028 


0.0328 


-0.0870 


0.9310 


# of counseling sessions in 1991-92 


-0.0044 


0.0031 


-1.3850 


0.1660 


# of counseling sessions in 1992-93 


-0.0052 


0.0030 


-1.7260 


0.0840 


# of counseling sessions in 1993-94 


0.0162 


0.0035 


4.6120 


0.0000 


# of tutoring sessions in 1991-92 


0.0030 


0.0022 


1.3730 


0.1700 


# of tutoring sessions in 1992-93 


-0.0043 


0.0023 


-1.8530 


0.0630 


# of tutoring sessions in 1993-94 


0.0078 


0.0024 


3.2140 


0.0020 


Some tutoring in years 4-6 


-0.0030 


0.0045 


-0.6820 


0.4950 


Some counseling in years 4-6 


0.0303 


0.0051 


5.9350 


0.0000 


Some services for disabled in years 4-6 


0.0233 


0.0647 


0.3600 


0.7190 


Field trip in years 4-6 


0.1411 


0.0368 


3.8400 


0.0000 


Contact with support services in years 4-6 


0.0064 


0.0042 


1.5150 


0.1300 


Missing data on high school GPA 


0.0741 


0.0676 


1.0970 


0.2730 
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Missing data on developmental courses 


0.0304 


0.0798 


0.3810 


0.7030 


Missing data on first generation 


-0.0722 


0.0465 


-1.5530 


0.1200 


Missing data on income 


-0.0016 


0.0344 


-0.0450 


0.9640 


Missing data on finishing tasks 


-0.0670 


0.0665 


-1.0070 


0.3140 


Missing data on B32 


-0.0618 


0.0673 


-0.9190 


0.3580 


Missing data on services for limited-English 


-0.0977 


0.1664 


-0.5870 


0.5570 


Missing data on student orientation 


0.0671 


0.1350 


0.4970 


0.6180 


Missing data on reentrance counseling 


0.0183 


0.0668 


0.2730 


0.7850 


Missing data on classroom instr.— basic skills 


-0.0155 


0.1661 


-0.0930 


0.9260 


Missing data on tutoring in 1991-92 


-0.0855 


0.0704 


-1.2150 


0.2250 


Missing data on tutoring in years 4-6 


0.1713 


0.1432 


1.1970 


0.2320 


Missing data on counseling in years 4-6 


-0.1320 


0.0814 


-1.6210 


0.1050 


Missing data on services for disabled in years 4-6 


-0.1946 


0.1608 


-1.2100 


0.2270 


Missing data on field trip in years 4-6 


0.3026 


0.2087 


1.4500 


0.1470 


Missing data on contact with support services in yrs. 
4-6 


-0.2193 


0.2083 


-1.0530 


0.2930 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 15. HLM analysis to predict cumulative GPAs six years after freshman entry in 



1991-92 using continuous measures of participation and adding propensity 
scores 



Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


2.4096 


0.0251 


95.9680 


0.0000 


SSS services are home-based 


0.1059 


0.0677 


1.5640 


0.1250 


Two-year institution 


0.1204 


0.0635 


1.8960 


0.0650 


Doctorate-granting institution 


-0.1433 


0.0938 


-1.5280 


0.1340 


No SSS programs at institution 


0.0283 


0.0682 


0.4140 


0.6810 


SSS services are blended 


-0.0524 


0.0697 


-0.7510 


0.4570 


Age in 1991 


0.0274 


0.0020 


13.3710 


0.0000 


Black student 


-0.3484 


0.0374 


-9.3130 


0.0000 


Hispanic student 


-0.1754 


0.0655 


-2.6760 


0.0080 


American Indian student 


-0.2598 


0.0900 


-2.8880 


0.0040 


Asian student 


-0.0920 


0.0662 


-1.3890 


0.1650 


Female student 


0.1384 


0.0220 


6.2780 


0.0000 


High school GPA 


0.2885 


0.0199 


14.5140 


0.0000 


Percentile on SAT/ACT 


0.2007 


0.0549 


3.6570 


0.0000 


Took any developmental course 


-0.1403 


0.0235 


-5.9650 


0.0000 


Hours on school activities 


0.0035 


0.0007 


5.0620 


0.0000 


Attended college after first year 


0.7135 


0.0339 


21.0210 


0.0000 


No parental education beyond high school 


0.0208 


0.0242 


0.8620 


0.3890 


Family income greater than $20,000 


0.0279 


0.0265 


1.0550 


0.2920 


Above average academic ability 


0.1446 


0.0249 


5.8090 


0.0000 


Would attend free tutoring 


-0.0803 


0.0265 


-3.0320 


0.0030 


Feel comfortable on this campus 


0.0590 


0.0274 


2.1550 


0.0310 


Number of hours: instr. courses 


-0.0009 


0.0007 


-1.3460 


0.1780 


Number of hours: profl. tutoring 


0.0205 


0.0478 


0.4290 


0.6680 


Number of hours: peer tutoring 


-0.0005 


0.0027 


-0.1690 


0.8660 


Number of hours: profl. counseling 


0.0052 


0.0124 


0.4180 


0.6760 


Number of hours: peer counseling 


-0.0452 


0.0305 


-1.4850 


0.1370 


Number of hours: labs 


0.0096 


0.0040 


2.4360 


0.0150 


Number of hours: workshops 


0.0207 


0.0111 


1.8690 


0.0610 


Number of hours: cultural events 


-0.0255 


0.0459 


-0.5570 


0.5770 


Number of hours: handicapped srvc. 


0.0077 


0.0097 


0.7930 


0.4280 


Propensity to receive instr. courses 


0.0002 


0.0009 


0.1990 


0.8420 


Propensity to receive profl. tutoring 


0.0109 


0.0274 


0.3990 


0.6900 


Propensity to receive peer tutoring 


0.0010 


0.0027 


0.3710 


0.7100 


Propensity to receive profl. counseling 


-0.0109 


0.0270 


-0.4040 


0.6860 


Propensity to receive peer counseling 


0.0357 


0.0195 


1.8290 


0.0670 


Propensity to receive labs 


-0.0011 


0.0029 


-0.3660 


0.7140 


Propensity to receive workshops 


-0.0056 


0.0067 


-0.8390 


0.4010 


Propensity to receive cultural events 


0.0066 


0.0167 


0.3970 


0.6910 


Propensity to receive handicapped srvc. 


0.0031 


0.0028 


1.1160 


0.2650 


Services for physically disabled 


-0.0042 


0.0345 


-0.1210 


0.9040 
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Services for limited-English ability 


0.0272 


0.0605 


0.4490 


0.6530 


Student orientation 


-0.0403 


0.0296 


-1.3620 


0.1730 


College reentrance counseling 


-0.0479 


0.0292 


-1.6420 


0.1000 


Classroom instr.— basic skills 


-0.0570 


0.0359 


-1.5880 


0.1120 


Classroom instr.— dvlpmntl. English 


0.0005 


0.0409 


0.0130 


0.9900 


Classroom instr.— dvlpmntl. math 


-0.0014 


0.0362 


-0.0390 


0.9690 


Cultural enrichment activities 


0.0128 


0.0052 


2.4590 


0.0140 


Referrals to agencies/resources 


0.0607 


0.0286 


2.1210 


0.0340 


# of counseling sessions in 1991-92 


-0.0040 


0.0028 


-1.4600 


0.1440 


# of counseling sessions in 1992-93 


-0.0006 


0.0027 


-0.2210 


0.8250 


# of counseling sessions in 1993-94 


0.0084 


0.0027 


3.1170 


0.0020 


# of tutoring sessions in 1991-92 


-0.0012 


0.0016 


-0.7230 


0.4700 


# of tutoring sessions in 1992-93 


0.0004 


0.0017 


0.2310 


0.8180 


# of tutoring sessions in 1993-94 


0.0010 


0.0019 


0.5410 


0.5880 


Some tutoring in years 4-6 


-0.0111 


0.0034 


-3.2530 


0.0020 


Some counseling in years 4-6 


0.0114 


0.0037 


3.0550 


0.0030 


Some services for disabled in years 4-6 


-0.0625 


0.0544 


-1.1490 


0.2510 


Field trip in years 4-6 


0.1279 


0.0266 


4.8070 


0.0000 


Contact with support services in years 4-6 


0.0159 


0.0031 


5.1840 


0.0000 


Missing data on student age 


0.5994 


0.0871 


6.8820 


0.0000 


Missing data on student race 


0.0143 


0.0932 


0.1540 


0.8780 


Missing data on high school GPA 


0.7643 


0.0863 


8.8590 


0.0000 


Missing data on SAT/ACT 


0.0318 


0.0350 


0.9090 


0.3640 


Missing data on developmental courses 


-0.1736 


0.0812 


-2.1390 


0.0320 


Missing data for hours on school activities 


0.1479 


0.0584 


2.5330 


0.0120 


Missing data on first generation 


-0.0885 


0.0535 


-1.6540 


0.0980 


Missing data on income 


-0.0210 


0.0372 


-0.5650 


0.5710 


Missing data on perceived ability 


0.1212 


0.1069 


1.1340 


0.2570 


Missing data on attending free tutoring 


-0.0017 


0.1941 


-0.0090 


0.9930 


Missing data on comfort on campus 


-0.1926 


0.1933 


-0.9960 


0.3200 


Missing data on services in later years 


0.2334 


0.0688 


3.3920 


0.0010 


Missing data on services for limited-English 


-0.0591 


0.4906 


-0.1210 


0.9050 


Missing data on student orientation 


-0.0896 


0.1243 


-0.7210 


0.4710 


Missing data on reentrance counseling 


0.1698 


0.0723 


2.3500 


0.0190 


Missing data on classroom instr.— basic skills 


-0.1490 


0.1041 


-1.4310 


0.1520 


Missing data on cultural enrichment activities 


-0.3003 


0.2369 


-1.2680 


0.2050 


Missing data on referrals to agencies/resources 


0.4740 


0.5429 


0.8730 


0.3830 


Missing data on tutoring in 1991-92 


-0.0946 


0.0754 


-1.2540 


0.2100 


Missing data on tutoring in years 4-6 


-0.0940 


0.1313 


-0.7160 


0.4740 


Missing data on counseling in years 4-6 


-0.0617 


0.0883 


-0.6980 


0.4850 


Missing data on services for disabled in years 4-6 


-0.0701 


0.1301 


-0.5390 


0.5890 


Missing data on field trip in years 4-6 


0.4124 


0.1808 


2.2810 


0.0230 


Missing data on contact with support services in yrs. 
4-6 


-0.2307 


0.1762 


-1.3090 


0.1910 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 16. HLM analysis to predict the number of credits earned six years after 



freshman entry in 1991-92 using continuous measures of participation and 
adding propensity scores 



Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


76.1127 


1.8124 


41.9950 


0.0000 


SSS services are home-based 


4.5506 


4.8904 


0.9310 


0.3580 


Two-year institution 


-14.5028 


4.5225 


-3.2070 


0.0030 


Doctorate-granting institution 


10.5614 


6.1001 


1.7310 


0.0900 


No SSS programs at institution 


3.4131 


4.8785 


0.7000 


0.4880 


SSS services are blended 


-3.1692 


5.0891 


-0.6230 


0.5370 


Black student 


-5.6227 


2.2409 


-2.5090 


0.0120 


Hispanic student 


1.1283 


4.6401 


0.2430 


0.8080 


American Indian student 


-20.4032 


5.2886 


-3.8580 


0.0000 


Asian student 


4.2210 


4.1520 


1.0170 


0.3100 


Female student 


4.5071 


1.2706 


3.5470 


0.0010 


Student was full-time 


10.6999 


4.6545 


2.2990 


0.0220 


High school GPA 


10.5388 


1.1386 


9.2560 


0.0000 


Percentile on SAT/ACT 


10.0531 


3.1568 


3.1850 


0.0020 


Took any developmental course 


-5.7366 


1.3569 


-4.2280 


0.0000 


Hours on school activities 


0.1775 


0.0397 


4.4690 


0.0000 


Attended college after first year 


43.8911 


1.9326 


22.7110 


0.0000 


No parental education beyond high school 


-0.5416 


1.3889 


-0.3900 


0.6960 


Family income greater than $20,000 


-1.2707 


1.6757 


-0.7580 


0.4480 


Have major concerns about ed. finances 


-3.8501 


1.2561 


-3.0650 


0.0030 


Once I start something, I finish it 


3.7426 


1.4220 


2.6320 


0.0090 


Feel comfortable on this campus 


7.1031 


1.5924 


4.4600 


0.0000 


Number of hours: instr. courses 


-0.0339 


0.0406 


-0.8350 


0.4040 


Number of hours: profl. tutoring 


-2.9658 


2.6790 


-1.1070 


0.2690 


Number of hours: peer tutoring 


0.2252 


0.1579 


1.4270 


0.1540 


Number of hours: profl. counseling 


0.6705 


0.7233 


0.9270 


0.3540 


Number of hours: peer counseling 


0.4628 


1.7842 


0.2590 


0.7950 


Number of hours: labs 


0.1704 


0.2287 


0.7450 


0.4560 


Number of hours: workshops 


-0.0199 


0.6439 


-0.0310 


0.9760 


Number of hours: cultural events 


-3.9044 


2.6594 


-1.4680 


0.1420 


Number of hours: handicapped srvc. 


-0.0996 


0.5648 


-0.1760 


0.8600 


Propensity to receive instr. courses 


0.0626 


0.0491 


1.2730 


0.2030 


Propensity to receive profl. tutoring 


4.5855 


1.5797 


2.9030 


0.0040 


Propensity to receive peer tutoring 


-0.2307 


0.1380 


-1.6720 


0.0940 


Propensity to receive profl. counseling 


1.3736 


2.7236 


0.5040 


0.6140 


Propensity to receive peer counseling 


-0.7014 


1.1361 


-0.6170 


0.5370 


Propensity to receive labs 


-0.0658 


0.1668 


-0.3940 


0.6930 


Propensity to receive workshops 


0.1878 


0.3882 


0.4840 


0.6280 


Propensity to receive cultural events 


0.0710 


0.9661 


0.0740 


0.9420 


Propensity to receive handicapped srvc. 


-0.0636 


0.1636 


-0.3880 


0.6970 


Services for physically disabled 


1.7497 


1.9916 


0.8790 


0.3800 
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Services for limited-English ability 


0.5564 


3.4790 


0.1600 


0.8730 


Student orientation 


-0.9701 


1.7039 


-0.5690 


0.5690 


College reentrance counseling 


-3.6612 


1.6849 


-2.1730 


0.0300 


Classroom instr.— basic skills 


-5.0852 


2.0723 


-2.4540 


0.0140 


Classroom instr.— dvlpmntl. English 


1.3197 


2.3592 


0.5590 


0.5750 


Classroom instr.— dvlpmntl. math 


-2.2437 


2.0880 


-1.0750 


0.2830 


Cultural enrichment activities 


0.7130 


0.2992 


2.3830 


0.0170 


Referrals to agencies/resources 


-2.0481 


1.6587 


-1.2350 


0.2170 


# of counseling sessions in 1991-92 


-0.1042 


0.1588 


-0.6560 


0.5110 


# of counseling sessions in 1992-93 


0.0058 


0.1559 


0.0370 


0.9710 


# of counseling sessions in 1993-94 


0.5198 


0.1565 


3.3210 


0.0010 


# of tutoring sessions in 1991-92 


-0.1417 


0.0924 


-1.5330 


0.1250 


# of tutoring sessions in 1992-93 


0.5115 


0.0976 


5.2380 


0.0000 


# of tutoring sessions in 1993-94 


0.2977 


0.1107 


2.6890 


0.0080 


Some tutoring in years 4-6 


0.0301 


0.1982 


0.1520 


0.8800 


Some counseling in years 4-6 


1.4482 


0.2171 


6.6710 


0.0000 


Some services for disabled in years 4-6 


-11.6425 


3.1724 


-3.6700 


0.0000 


Field trip in years 4-6 


13.0440 


1.5407 


8.4660 


0.0000 


Contact with support services in years 4-6 


2.1016 


0.1777 


11.8280 


0.0000 


Missing data on student race 


-0.7630 


5.8918 


-0.1300 


0.8970 


Missing data on full-time status 


19.4546 


7.5037 


2.5930 


0.0100 


Missing data on high school GPA 


27.2416 


4.9852 


5.4650 


0.0000 


Missing data on SAT/ACT 


-0.0586 


1.9861 


-0.0290 


0.9770 


Missing data on developmental courses 


-1.3646 


4.6997 


-0.2900 


0.7710 


Missing data for hours on school activities 


5.4407 


3.5825 


1.5190 


0.1290 


Missing data on first generation 


-2.5116 


3.1752 


-0.7910 


0.4290 


Missing data on income 


-0.3377 


2.2521 


-0.1500 


0.8810 


Missing data on perceived ability 


0.6781 


6.2966 


0.1080 


0.9150 


Missing data on financial concerns 


-9.5040 


5.7126 


-1.6640 


0.0960 


Missing data on finishing tasks 


10.2830 


6.1636 


1.6680 


0.0950 


Missing data on comfort on campus 


-5.1408 


7.6061 


-0.6760 


0.4990 


Missing data on services in later years 


-1.6456 


3.9744 


-0.4140 


0.6780 


Missing data on services for limited-English 


-17.7201 


28.3003 


-0.6260 


0.5310 


Missing data on student orientation 


0.5878 


7.1507 


0.0820 


0.9350 


Missing data on reentrance counseling 


0.5508 


4.1750 


0.1320 


0.8950 


Missing data on classroom instr.— basic skills 


6.2460 


5.8836 


1.0620 


0.2890 


Missing data on cultural enrichment activities 


-30.0858 


13.6867 


-2.1980 


0.0280 


Missing data on referrals to agencies/resources 


42.7787 


31.3177 


1.3660 


0.1720 


Missing data on tutoring in 1991-92 


-4.6920 


4.3594 


-1.0760 


0.2820 


Missing data on tutoring in years 4-6 


0.1428 


7.6608 


0.0190 


0.9850 


Missing data on counseling in years 4-6 


-5.5397 


5.1007 


-1.0860 


0.2780 


Missing data on services for disabled in years 4-6 


-8.8703 


7.5542 


-1.1740 


0.2410 


Missing data on field trip in years 4-6 


7.4848 


10.6760 


0.7010 


0.4830 


Missing data on contact with support services in yrs. 4-6 


6.0580 


10.3757 


0.5840 


0.5590 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 17. HLM analysis to predict retention or baccalaureate degree completion at the 
same institution six years after freshman entry in 1991-92 using continuous 
measures of participation and adding propensity scores 


Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.3182 


0.0111 


28.6840 


0.0000 


SSS services are home-based 


0.0632 


0.0298 


2.1220 


0.0400 


Two-year institution 


-0.2221 


0.0288 


-7.7180 


0.0000 


Doctorate-granting institution 


0.1084 


0.0466 


2.3240 


0.0250 


No SSS programs at institution 


0.0590 


0.0308 


1.9140 


0.0620 


SSS services are blended 


0.0603 


0.0300 


2.0070 


0.0510 


Female student 


0.0368 


0.0130 


2.8400 


0.0050 


High school GPA 


0.0623 


0.0112 


5.5550 


0.0000 


Percentile on SAT/ACT 


0.1144 


0.0319 


3.5890 


0.0010 


Took any developmental course 


-0.0506 


0.0139 


-3.6500 


0.0000 


Attended college after first year 


0.1640 


0.0198 


8.2930 


0.0000 


No parental education beyond high school 


0.0048 


0.0140 


0.3440 


0.7310 


Family income greater than $20,000 


-0.0124 


0.0154 


-0.8050 


0.4210 


Feel comfortable on this campus 


0.0683 


0.0160 


4.2610 


0.0000 


Number of hours: instr. courses 


0.0001 


0.0004 


0.1730 


0.8630 


Number of hours: profl. tutoring 


0.0146 


0.0275 


0.5330 


0.5940 


Number of hours: peer tutoring 


0.0019 


0.0016 


1.1990 


0.2310 


Number of hours: profl. counseling 


0.0009 


0.0072 


0.1230 


0.9030 


Number of hours: peer counseling 


0.0004 


0.0176 


0.0210 


0.9830 


Number of hours: labs 


0.0010 


0.0023 


0.4470 


0.6540 


Number of hours: workshops 


-0.0110 


0.0065 


-1.6910 


0.0900 


Number of hours: cultural events 


-0.0020 


0.0269 


-0.0750 


0.9410 


Number of hours: handicapped srvc 


0.0067 


0.0057 


1.1690 


0.2430 


Propensity to receive instr. courses 


0.0008 


0.0004 


1.8690 


0.0610 


Propensity to receive profl. tutoring 


0.0122 


0.0153 


0.8000 


0.4240 


Propensity to receive peer tutoring 


-0.0009 


0.0014 


-0.6850 


0.4930 


Propensity to receive profl. counseling 


0.0128 


0.0117 


1.0990 


0.2720 


Propensity to receive peer counseling 


-0.0119 


0.0106 


-1.1140 


0.2660 


Propensity to receive labs 


-0.0009 


0.0016 


-0.5600 


0.5750 


Propensity to receive workshops 


0.0029 


0.0038 


0.7770 


0.4370 


Propensity to receive cultural events 


0.0072 


0.0096 


0.7430 


0.4570 


Propensity to receive handicapped srvc. 


0.0002 


0.0016 


0.1380 


0.8910 


Services for physically disabled 


0.0279 


0.0205 


1.3630 


0.1730 


Services for limited-English ability 


0.0270 


0.0356 


0.7600 


0.4470 


Student orientation 


0.0068 


0.0174 


0.3910 


0.6960 


College reentrance counseling 


-0.0100 


0.0173 


-0.5810 


0.5610 


Classroom instr.— basic skills 


-0.0162 


0.0213 


-0.7600 


0.4470 


Classroom instr.— dvlpmntl. English 


0.0053 


0.0242 


0.2210 


0.8250 


Classroom instr.— dvlpmntl. math 


-0.0063 


0.0214 


-0.2930 


0.7700 


Cultural enrichment activities 


-0.0008 


0.0031 


-0.2450 


0.8070 


Referrals to agencies/resources 


-0.0225 


0.0169 


-1.3290 


0.1840 



Continues to next page. 



102 










Continued from previous page. 



# of counseling sessions in 1991-92 


-0.0021 


0.0016 


-1.2870 


0.1980 


# of counseling sessions in 1992-93 


-0.0009 


0.0016 


-0.5480 


0.5830 


# of counseling sessions in 1993-94 


0.0038 


0.0016 


2.3700 


0.0180 


# of tutoring sessions in 1991-92 


-0.0014 


0.0009 


-1.5020 


0.1330 


# of tutoring sessions in 1992-93 


0.0020 


0.0010 


1.9940 


0.0460 


# of tutoring sessions in 1993-94 


0.0024 


0.0011 


2.1430 


0.0320 


Some tutoring in years 4-6 


0.0016 


0.0020 


0.7870 


0.4310 


Some counseling in years 4-6 


0.0110 


0.0022 


4.9310 


0.0000 


Some services for disabled in years 4-6 


-0.0597 


0.0321 


-1.8600 


0.0620 


Field trip in years 4-6 


0.0737 


0.0158 


4.6660 


0.0000 


Contact with support services in years 4-6 


0.0176 


0.0018 


9.6740 


0.0000 


Missing data on high school GPA 


0.2100 


0.0482 


4.3550 


0.0000 


Missing data on SAT/ACT 


0.0343 


0.0200 


1.7160 


0.0860 


Missing data on developmental courses 


-0.1286 


0.0481 


-2.6750 


0.0080 


Missing data for hours on school activities 


-0.0081 


0.0320 


-0.2520 


0.8010 


Missing data on first generation 


0.0302 


0.0308 


0.9800 


0.3270 


Missing data on income 


-0.0038 


0.0216 


-0.1760 


0.8610 


Missing data on comfort on campus 


-0.0010 


0.0520 


-0.0190 


0.9850 


Missing data on services in later years 


0.0610 


0.0407 


1.4990 


0.1340 


Missing data on services for limited-English 


-0.4886 


0.2921 


-1.6730 


0.0940 


Missing data on student orientation 


0.0232 


0.0718 


0.3230 


0.7460 


Missing data on reentrance counseling 


0.0197 


0.0423 


0.4660 


0.6410 


Missing data on classroom instr.— basic skills 


0.0296 


0.0601 


0.4930 


0.6220 


Missing data on cultural enrichment activities 


0.2612 


0.1408 


1.8560 


0.0630 


Missing data on referrals to agencies/resources 


0.1782 


0.3231 


0.5520 


0.5810 


Missing data on tutoring in 1991-92 


0.0069 


0.0447 


0.1550 


0.8770 


Missing data on tutoring in years 4-6 


-0.0046 


0.0768 


-0.0600 


0.9530 


Missing data on counseling in years 4-6 


-0.0517 


0.0524 


-0.9870 


0.3240 


Missing data on services for disabled in years 4-6 


-0.0082 


0.0774 


-0.1060 


0.9160 


Missing data on field trip in years 4-6 


0.0505 


0.1072 


0.4710 


0.6370 


Missing data on contact with support services in yrs. 4-6 


0.0119 


0.1048 


0.1130 


0.9100 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 18. HLM analysis to predict retention or baccalaureate degree completion at any 
institution six years after freshman entry in 1991-92 using continuous 
measures of participation and adding propensity scores 


Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.6196 


0.0099 


62.6400 


0.0000 


SSS services are home-based 


0.0766 


0.0266 


2.8760 


0.0070 


Two-year institution 


-0.1315 


0.0262 


-5.0220 


0.0000 


Doctorate-granting institution 


0.1365 


0.0437 


3.1240 


0.0040 


No SSS programs at institution 


0.0675 


0.0278 


2.4250 


0.0200 


SSS services are blended 


0.0437 


0.0264 


1.6580 


0.1050 


Female student 


0.0540 


0.0132 


4.0980 


0.0000 


High school GPA 


0.0426 


0.0115 


3.6920 


0.0000 


Percentile on SAT/ACT 


0.0939 


0.0319 


2.9410 


0.0040 


Took any developmental course 


-0.0525 


0.0138 


-3.8140 


0.0000 


Hours on school activities 


0.0005 


0.0004 


1.1720 


0.2420 


Attended college after first year 


0.3501 


0.0234 


14.9680 


0.0000 


No parental education beyond high school 


-0.0508 


0.0141 


-3.5960 


0.0010 


Family income greater than $20,000 


0.0114 


0.0155 


0.7340 


0.4630 


Above average academic ability 


0.0211 


0.0139 


1.5180 


0.1290 


Number of hours: instr. courses 


-0.0006 


0.0004 


-1.5740 


0.1150 


Number of hours: profl. tutoring 


0.0079 


0.0284 


0.2780 


0.7810 


Number of hours: peer tutoring 


0.0010 


0.0016 


0.6170 


0.5370 


Number of hours: profl. counseling 


0.0089 


0.0071 


1.2500 


0.2120 


Number of hours: peer counseling 


-0.0008 


0.0173 


-0.0440 


0.9650 


Number of hours: labs 


0.0009 


0.0024 


0.3630 


0.7160 


Number of hours: workshops 


-0.0070 


0.0064 


-1.0960 


0.2740 


Number of hours: cultural events 


-0.0219 


0.0267 


-0.8200 


0.4120 


Number of hours: handicapped srvc. 


-0.0007 


0.0059 


-0.1170 


0.9070 


Propensity to receive instr. courses 


0.0015 


0.0004 


3.4700 


0.0010 


Propensity to receive profl. tutoring 


0.0224 


0.0166 


1.3480 


0.1780 


Propensity to receive peer tutoring 


-0.0005 


0.0014 


-0.3500 


0.7260 


Propensity to receive profl. counseling 


-0.0121 


0.0125 


-0.9750 


0.3300 


Propensity to receive peer counseling 


0.0070 


0.0106 


0.6670 


0.5050 


Propensity to receive labs 


-0.0016 


0.0017 


-0.9350 


0.3500 


Propensity to receive workshops 


-0.0018 


0.0037 


-0.4750 


0.6340 


Propensity to receive cultural events 


-0.0074 


0.0095 


-0.7810 


0.4350 


Propensity to receive handicapped srvc. 


0.0002 


0.0016 


0.1140 


0.9100 


Services for physically disabled 


-0.0090 


0.0231 


-0.3910 


0.6960 


Services for limited-English ability 


0.0490 


0.0374 


1.3110 


0.1900 


Student orientation 


0.0143 


0.0177 


0.8100 


0.4180 


College reentrance counseling 


0.0123 


0.0191 


0.6440 


0.5190 


Classroom instr.— basic skills 


-0.0059 


0.0216 


-0.2740 


0.7840 


Classroom instr.— dvlpmntl. English 


-0.0221 


0.0243 


-0.9080 


0.3640 


Classroom instr.— dvlpmntl. math 


-0.0166 


0.0217 


-0.7660 


0.4440 


Cultural enrichment activities 


0.0029 


0.0035 


0.8380 


0.4020 
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Referrals to agencies/resources 


-0.0340 


0.0171 


-1.9940 


0.0460 


# of counseling sessions in 1991-92 


-0.0027 


0.0016 


-1.6500 


0.0990 


# of counseling sessions in 1992-93 


-0.0020 


0.0015 


-1.3150 


0.1890 


# of counseling sessions in 1993-94 


0.0034 


0.0015 


2.1990 


0.0280 


# of tutoring sessions in 1991-92 


-0.0016 


0.0009 


-1.6800 


0.0930 


# of tutoring sessions in 1992-93 


0.0008 


0.0010 


0.8440 


0.3990 


# of tutoring sessions in 1993-94 


0.0048 


0.0011 


4.3290 


0.0000 


Some tutoring in years 4-6 


-0.0016 


0.0019 


-0.8320 


0.4060 


Some counseling in years 4-6 


0.0208 


0.0021 


9.9940 


0.0000 


Some services for disabled in years 4-6 


-0.0191 


0.0299 


-0.6390 


0.5230 


Field trip in years 4-6 


0.1320 


0.0147 


8.9590 


0.0000 


Contact with support services in years 4-6 


0.0194 


0.0017 


11.4010 


0.0000 


Missing data on fulltime status 


-0.0055 


0.0673 


-0.0820 


0.9350 


Missing data on high school GPA 


0.1319 


0.0486 


2.7170 


0.0070 


Missing data on SAT/ACT 


-0.0172 


0.0202 


-0.8500 


0.3950 


Missing data on developmental courses 


0.0418 


0.0541 


0.7730 


0.4390 


Missing data for hours on school activities 


0.0023 


0.0385 


0.0590 


0.9530 


Missing data on first generation 


-0.0442 


0.0343 


-1.2890 


0.1980 


Missing data on income 


-0.0028 


0.0222 


-0.1250 


0.9010 


Missing data on perceived ability 


0.0566 


0.0569 


0.9940 


0.3210 


Missing data on services in later years 


-0.0473 


0.0468 


-1.0110 


0.3120 


Missing data on services for limited-English 


-0.1851 


0.2720 


-0.6800 


0.4960 


Missing data on student orientation 


0.0319 


0.0744 


0.4290 


0.6670 


Missing data on reentrance counseling 


0.0089 


0.0413 


0.2150 


0.8300 


Missing data on classroom instr.— basic skills 


-0.0155 


0.0602 


-0.2580 


0.7970 


Missing data on cultural enrichment activities 


-0.2483 


0.1314 


-1.8900 


0.0580 


Missing data on referrals to agencies/resources 


0.3955 


0.3010 


1.3140 


0.1890 


Missing data on tutoring in 1991-92 


-0.1117 


0.0572 


-1.9550 


0.0500 


Missing data on tutoring in years 4-6 


0.2011 


0.0720 


2.7940 


0.0060 


Missing data on counseling in years 4-6 


-0.0378 


0.0488 


-0.7750 


0.4380 


Missing data on services for disabled in years 4-6 


-0.1129 


0.0720 


-1.5690 


0.1160 


Missing data on field trip in years 4-6 


0.2226 


0.0997 


2.2330 


0.0250 


Missing data on contact with support services in yrs. 4-6 


0.1243 


0.0975 


1.2740 


0.2030 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A- 19. HLM analysis to predict baccalaureate degree completion (or higher) at any 
institution six years after freshman entry in 1991-92 using continuous 



measures of participation and adding propensity scores 


Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 



Intercept 

SSS services are home-based 
Two-year institution 
Doctorate-granting institution 
No SSS programs at institution 
SSS services are blended 
Black student 
Hispanic student 
American Indian student 
Asian student 
Female student 

Live in college housing or frat./sorority 
High school GPA 
Percentile on SAT/ACT 
Took any developmental course 
Hours on school activities 
Attended college after first year 
No parental education beyond high school 
Family income greater than $20,000 
Have major concerns about ed. finances 
Once I start something, I finish it 
Number of hours: instr. courses 
Number of hours: profl. tutoring 
Number of hours: peer tutoring 
Number of hours: profl. counseling 
Number of hours: peer counseling 
Number of hours: labs 
Number of hours: workshops 
Number of hours: cultural events 
Number of hours: handicapped srvc. 
Propensity to receive instr. courses 
Propensity to receive profl. tutoring 
Propensity to receive peer tutoring 
Propensity to receive profl. counseling 
Propensity to receive peer counseling 
Propensity to receive labs 
Propensity to receive workshops 
Propensity to receive cultural events 
Propensity to receive handicapped srvc 
Services for physically disabled 
Services for limited-English ability 



0.3985 


0.0143 


27.9370 


0.0000 


0.0452 


0.0386 


1.1710 


0.2490 


-0.1713 


0.0365 


-4.6890 


0.0000 


0.1128 


0.0542 


2.0810 


0.0430 


0.0415 


0.0390 


1.0630 


0.2940 


0.0228 


0.0394 


0.5790 


0.5650 


-0.0311 


0.0231 


-1.3470 


0.1780 


0.0275 


0.0393 


0.6980 


0.4850 


-0.1825 


0.0559 


-3.2670 


0.0010 


0.1006 


0.0417 


2.4150 


0.0160 


0.0344 


0.0138 


2.4920 


0.0130 


0.0466 


0.0222 


2.1000 


0.0350 


0.0980 


0.0121 


8.0870 


0.0000 


0.1359 


0.0335 


4.0630 


0.0000 


-0.0889 


0.0143 


-6.2070 


0.0000 


0.0009 


0.0004 


2.1000 


0.0350 


0.1436 


0.0242 


5.9330 


0.0000 


-0.0253 


0.0150 


-1.6840 


0.0920 


0.0305 


0.0163 


1.8750 


0.0600 


-0.0418 


0.0136 


-3.0770 


0.0030 


0.0434 


0.0153 


2.8480 


0.0050 


0.0000 


0.0004 


0.0590 


0.9540 


-0.0236 


0.0297 


-0.7940 


0.4270 


0.0019 


0.0017 


1.0980 


0.2720 


0.0025 


0.0077 


0.3210 


0.7480 


-0.0099 


0.0188 


-0.5270 


0.5980 


0.0031 


0.0026 


1.2090 


0.2270 


-0.0064 


0.0068 


-0.9470 


0.3440 


-0.0426 


0.0281 


-1.5200 


0.1280 


0.0065 


0.0062 


1.0510 


0.2940 


0.0008 


0.0005 


1.4430 


0.1490 


0.0270 


0.0183 


1.4780 


0.1390 


-0.0024 


0.0015 


-1.5940 


0.1110 


0.0392 


0.0156 


2.5200 


0.0120 


0.0100 


0.0121 


0.8250 


0.4090 


-0.0018 


0.0018 


-0.9690 


0.3330 


-0.0019 


0.0045 


-0.4280 


0.6680 


0.0165 


0.0102 


1.6130 


0.1060 


-0.0028 


0.0017 


-1.6050 


0.1080 


0.0161 


0.0239 


0.6730 


0.5010 


0.0055 


0.0392 


0.1400 


0.8890 
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Student orientation 


0.0007 


0.0186 


0.0380 


0.9700 


College reentrance counseling 


-0.0130 


0.0198 


-0.6580 


0.5100 


Classroom instr.— basic skills 


0.0000 


0.0225 


-0.0010 


0.9990 


Classroom instr.— dvlpmntl. English 


-0.0014 


0.0255 


-0.0550 


0.9560 


Classroom instr.— dvlpmntl. math 


-0.0146 


0.0227 


-0.6410 


0.5220 


Cultural enrichment activities 


-0.0023 


0.0036 


-0.6470 


0.5180 


Referrals to agencies/resources 


-0.0066 


0.0178 


-0.3700 


0.7110 


# of counseling sessions in 1991-92 


-0.0029 


0.0017 


-1.7140 


0.0860 


# of counseling sessions in 1992-93 


0.0017 


0.0016 


1.0540 


0.2920 


# of counseling sessions in 1993-94 


0.0041 


0.0016 


2.6240 


0.0090 


# of tutoring sessions in 1991-92 


0.0000 


0.0010 


0.0470 


0.9630 


# of tutoring sessions in 1992-93 


0.0013 


0.0010 


1.2830 


0.2000 


# of tutoring sessions in 1993-94 


0.0005 


0.0012 


0.4490 


0.6530 


Some tutoring in years 4-6 


-0.0040 


0.0020 


-2.0250 


0.0430 


Some counseling in years 4-6 


0.0117 


0.0022 


5.4390 


0.0000 


Some services for disabled in years 4-6 


-0.1063 


0.0309 


-3.4350 


0.0010 


Field trip in years 4-6 


0.1334 


0.0152 


8.7810 


0.0000 


Contact with support services in years 4-6 


0.0162 


0.0018 


9.2270 


0.0000 


Missing data on student age 


0.0582 


0.0568 


1.0240 


0.3060 


Missing data on student race 


0.1168 


0.0790 


1.4790 


0.1390 


Missing data on high school GPA 


0.3029 


0.0521 


5.8140 


0.0000 


Missing data on SAT/ACT 


-0.0046 


0.0213 


-0.2170 


0.8280 


Missing data on developmental courses 


-0.0250 


0.0576 


-0.4340 


0.6640 


Missing data for hours on school activities 


0.0389 


0.0402 


0.9680 


0.3340 


Missing data on first generation 


-0.0496 


0.0360 


-1.3780 


0.1680 


Missing data on income 


0.0444 


0.0234 


1.8990 


0.0570 


Missing data on financial concerns 


-0.0480 


0.0648 


-0.7410 


0.4590 


Missing data on finishing tasks 


-0.0245 


0.0567 


-0.4330 


0.6640 


Missing data on B32 


0.0099 


0.0504 


0.1970 


0.8440 


Missing data on services for limited-English 


0.4974 


0.2802 


1.7750 


0.0750 


Missing data on student orientation 


0.0115 


0.0771 


0.1490 


0.8820 


Missing data on reentrance counseling 


0.0258 


0.0429 


0.6020 


0.5470 


Missing data on classroom instr.— basic skills 


0.0037 


0.0621 


0.0600 


0.9530 


Missing data on cultural enrichment activities 


-0.1003 


0.1359 


-0.7380 


0.4600 


Missing data on referrals to agencies/resources 


-0.4757 


0.3101 


-1.5340 


0.1250 


Missing data on tutoring in 1991-92 


-0.0910 


0.0617 


-1.4760 


0.1400 


Missing data on tutoring in years 4-6 


0.1561 


0.0747 


2.0890 


0.0360 


Missing data on counseling in years 4-6 


0.0357 


0.0503 


0.7090 


0.4780 


Missing data on services for disabled in years 4-6 


-0.1335 


0.0742 


-1.7990 


0.0720 


Missing data on field trip in years 4-6 


0.3262 


0.1032 


3.1600 


0.0020 


Missing data on contact with support services in yrs. 
4-6 


0.0886 


0.1007 


0.8800 


0.3790 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Exhibit A-20. HLM analysis to predict associate’s degree completion (or higher) at any 



institution six years after freshman entry in 1991-92 using continuous 
measures of participation and adding propensity scores 



Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.5382 


0.0153 


35.2720 


0.0000 


SSS services are home-based 


0.0238 


0.0411 


0.5780 


0.5660 


Two-year institution 


-0.0041 


0.0387 


-0.1070 


0.9160 


Doctorate-granting institution 


0.0596 


0.0582 


1.0240 


0.3120 


No SSS programs at institution 


0.0073 


0.0417 


0.1750 


0.8620 


SSS services are blended 


-0.0127 


0.0422 


-0.3000 


0.7650 


Black student 


-0.1138 


0.0286 


-3.9750 


0.0000 


Hispanic student 


-0.1422 


0.0775 


-1.8340 


0.0660 


American Indian student 


-0.2444 


0.0611 


-3.9990 


0.0000 


Asian student 


-0.0278 


0.0566 


-0.4910 


0.6230 


Female student 


0.0573 


0.0150 


3.8240 


0.0000 


Student was full-time 


0.1766 


0.0790 


2.2360 


0.0250 


High school GPA 


0.1005 


0.0141 


7.1390 


0.0000 


Took any developmental course 


-0.0891 


0.0153 


-5.8200 


0.0000 


Attended college after first year 


0.2774 


0.0263 


10.5680 


0.0000 


No parental education beyond high school 


-0.0183 


0.0162 


-1.1270 


0.2600 


Family income greater than $20,000 


-0.0030 


0.0217 


-0.1360 


0.8920 


Above average academic ability 


0.0468 


0.0238 


1.9690 


0.0490 


Have major concerns about ed. finances 


-0.0371 


0.0147 


-2.5150 


0.0120 


Number of hours: instr. courses 


-0.0002 


0.0005 


-0.3400 


0.7340 


Number of hours: profl. tutoring 


-0.0324 


0.0322 


-1.0050 


0.3150 


Number of hours: peer tutoring 


0.0012 


0.0018 


0.6530 


0.5130 


Number of hours: profl. counseling 


-0.0008 


0.0083 


-0.1020 


0.9190 


Number of hours: peer counseling 


0.0050 


0.0205 


0.2450 


0.8070 


Number of hours: labs 


-0.0031 


0.0028 


-1.1110 


0.2670 


Number of hours: workshops 


-0.0074 


0.0073 


-1.0120 


0.3120 


Number of hours: cultural events 


0.0067 


0.0302 


0.2210 


0.8250 


Number of hours: handicapped srvc. 


0.0084 


0.0067 


1.2500 


0.2120 


Propensity to receive instr. courses 


0.0001 


0.0006 


0.1890 


0.8500 


Propensity to receive profl. tutoring 


0.0238 


0.0197 


1.2120 


0.2260 


Propensity to receive peer tutoring 


-0.0028 


0.0016 


-1.7080 


0.0870 


Propensity to receive profl. counseling 


-0.0555 


0.0500 


-1.1090 


0.2680 


Propensity to receive peer counseling 


0.0141 


0.0130 


1.0860 


0.2780 


Propensity to receive labs 


-0.0003 


0.0020 


-0.1580 


0.8750 


Propensity to receive workshops 


0.0032 


0.0044 


0.7290 


0.4660 


Propensity to receive cultural events 


0.0099 


0.0113 


0.8780 


0.3800 


Propensity to receive handicapped srvc. 


-0.0013 


0.0018 


-0.7300 


0.4660 


Services for physically disabled 


0.0032 


0.0253 


0.1270 


0.8990 


Services for limited-English ability 


0.0098 


0.0423 


0.2310 


0.8170 


Student orientation 


0.0158 


0.0201 


0.7840 


0.4330 


College reentrance counseling 


-0.0063 


0.0215 


-0.2920 


0.7700 
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Classroom instr.— basic skills 


-0.0139 


0.0244 


-0.5710 


Classroom instr.— dvlpmntl. English 


-0.0093 


0.0276 


-0.3370 


Classroom instr.— dvlpmntl. math 


-0.0222 


0.0247 


-0.9000 


Cultural enrichment activities 


-0.0015 


0.0038 


-0.4040 


Referrals to agencies/resources 


0.0118 


0.0194 


0.6080 


# of counseling sessions in 1991-92 


-0.0018 


0.0019 


-0.9760 


# of counseling sessions in 1992-93 


0.0005 


0.0017 


0.2580 


# of counseling sessions in 1993-94 


0.0053 


0.0017 


3.0440 


# of tutoring sessions in 1991-92 


0.0000 


0.0011 


0.0020 


# of tutoring sessions in 1992-93 


0.0016 


0.0011 


1.4080 


# of tutoring sessions in 1993-94 


0.0003 


0.0013 


0.2320 


Some tutoring in years 4-6 


-0.0048 


0.0021 


-2.2590 


Some counseling in years 4-6 


0.0087 


0.0023 


3.7000 


Some services for disabled in years 4-6 


-0.1038 


0.0338 


-3.0750 


Field trip in years 4-6 


0.1184 


0.0166 


7.1450 


Contact with support services in years 4-6 


0.0178 


0.0019 


9.2790 


Missing data on student race 


-0.1322 


0.0795 


-1.6630 


Missing data on fulltime status 


0.1719 


0.0883 


1.9470 


Missing data on high school GPA 


0.2664 


0.0602 


4.4270 


Missing data on developmental courses 


-0.0948 


0.0627 


-1.5120 


Missing data on first generation 


-0.0832 


0.0399 


-2.0880 


Missing data on income 


0.0027 


0.0289 


0.0940 


Missing data on perceived ability 


0.0888 


0.0729 


1.2170 


Missing data on financial concerns 


0.0077 


0.0703 


0.1090 


Missing data on services in later years 


0.0210 


0.0539 


0.3890 


Missing data on services for limited-English 


0.1387 


0.3061 


0.4530 


Missing data on student orientation 


0.0084 


0.0836 


0.1000 


Missing data on reentrance counseling 


0.0045 


0.0466 


0.0970 


Missing data on classroom instr.— basic skills 


0.0210 


0.0677 


0.3100 


Missing data on cultural enrichment activities 


-0.1688 


0.1482 


-1.1380 


Missing data on referrals to agencies/resources 


-0.0264 


0.3387 


-0.0780 


Missing data on tutoring in 1991-92 


-0.0254 


0.0672 


-0.3780 


Missing data on tutoring in years 4-6 


0.1311 


0.0820 


1.5990 


Missing data on counseling in years 4-6 


0.0627 


0.0549 


1.1410 


Missing data on services for disabled in years 4-6 


0.0034 


0.0811 


0.0420 


Missing data on field trip in years 4-6 


0.1918 


0.1125 


1.7050 


Missing data on contact with support services in yrs. 4-6 


0.0721 


0.1099 


0.6570 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 



0.5680 

0.7360 

0.3680 

0.6860 

0.5430 

0.3300 

0.7960 

0.0030 

0.9990 

0.1590 

0.8170 

0.0240 

0.0000 

0.0030 

0.0000 

0.0000 

0.0960 

0.0510 

0.0000 

0.1300 

0.0360 

0.9250 

0.2240 

0.9140 

0.6970 

0.6500 

0.9210 

0.9230 

0.7560 

0.2550 

0.9380 

0.7050 

0.1100 

0.2540 

0.9670 

0.0880 

0.5110 
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Exhibit A-21. HLM analysis to predict transfers from two-year to four-year institutions six 



years after freshman entry in 1991-92 using continuous measures of 
participation and adding propensity scores 


Variable 


Coefficient 


Standard 

Error 


T-ratio 


P-value 


Intercept 


0.2186 


0.0122 


17.9280 


0.0000 


SSS services are home-based 


0.0375 


0.0447 


0.8410 


0.4170 


No SSS programs at institution 


0.1060 


0.0336 


3.1530 


0.0090 


SSS services are blended 


0.0285 


0.0354 


0.8050 


0.4370 


Female student 


-0.0704 


0.0251 


-2.8050 


0.0050 


High school GPA 


0.0629 


0.0178 


3.5340 


0.0010 


Took any developmental course 


-0.0609 


0.0252 


-2.4150 


0.0160 


Attended college after first year 


0.0930 


0.0295 


3.1510 


0.0020 


No parental education beyond high school 


-0.1195 


0.0271 


-4.4080 


0.0000 


Family income greater than $20,000 


0.0025 


0.0281 


0.0880 


0.9300 


Once I start something, I finish it 


-0.0320 


0.0262 


-1.2190 


0.2230 


Number of hours: instr. courses 


-0.0012 


0.0006 


-1.9150 


0.0550 


Number of hours: profl. tutoring 


-0.0167 


0.0589 


-0.2840 


0.7770 


Number of hours: peer tutoring 


0.0039 


0.0037 


1.0410 


0.2980 


Number of hours: profl. counseling 


0.0169 


0.0120 


1.4060 


0.1600 


Number of hours: peer counseling 


-0.0917 


0.1648 


-0.5560 


0.5780 


Number of hours: labs 


0.0040 


0.0027 


1.4680 


0.1420 


Number of hours: workshops 


0.0344 


0.0321 


1.0720 


0.2840 


Number of hours: cultural events 


0.1990 


0.1162 


1.7130 


0.0860 


Propensity to receive instr. courses 


0.0021 


0.0007 


3.1170 


0.0020 


Propensity to receive profl. tutoring 


0.0421 


0.0342 


1.2300 


0.2190 


Propensity to receive peer tutoring 


-0.0049 


0.0033 


-1.4830 


0.1380 


Propensity to receive profl. counseling 


0.0159 


0.0187 


0.8500 


0.3950 


Propensity to receive peer counseling 


-0.0461 


0.0642 


-0.7190 


0.4720 


Propensity to receive labs 


0.0000 


0.0021 


0.0130 


0.9890 


Propensity to receive workshops 


0.0355 


0.0139 


2.5540 


0.0110 


Propensity to receive cultural events 


-0.1200 


0.0584 


-2.0540 


0.0400 


Propensity to receive handicapped srvc. 


0.0027 


0.0068 


0.3940 


0.6940 


Services for physically disabled 


0.0465 


0.0312 


1.4910 


0.1360 


Services for limited-English ability 


-0.0761 


0.0570 


-1.3340 


0.1820 


Student orientation 


0.0055 


0.0254 


0.2160 


0.8290 


College reentrance counseling 


0.0486 


0.0257 


1.8890 


0.0590 


Classroom instr.— basic skills 


-0.0604 


0.0392 


-1.5410 


0.1230 


Classroom instr.— dvlpmntl. English 


0.0152 


0.0418 


0.3640 


0.7150 


Classroom instr.— dvlpmntl. math 


-0.0750 


0.0384 


-1.9540 


0.0500 


Cultural enrichment activities 


-0.0258 


0.0146 


-1.7640 


0.0770 


Referrals to agencies/resources 


-0.0028 


0.0329 


-0.0870 


0.9310 


# of counseling sessions in 1991-92 


-0.0033 


0.0032 


-1.0310 


0.3030 


# of counseling sessions in 1992-93 


-0.0072 


0.0030 


-2.3530 


0.0190 


# of counseling sessions in 1993-94 


0.0159 


0.0035 


4.5450 


0.0000 


# of tutoring sessions in 1991-92 


0.0034 


0.0021 


1.6050 


0.1080 
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# of tutoring sessions in 1992-93 


-0.0051 


0.0023 


-2.2700 


0.0230 


# of tutoring sessions in 1993-94 


0.0082 


0.0024 


3.3640 


0.0010 


Some tutoring in years 4-6 


-0.0030 


0.0044 


-0.6710 


0.5020 


Some counseling in years 4-6 


0.0289 


0.0051 


5.6540 


0.0000 


Some services for disabled in years 4-6 


-0.0181 


0.0618 


-0.2940 


0.7690 


Field trip in years 4-6 


0.1334 


0.0366 


3.6420 


0.0000 


Contact with support services in years 4-6 


0.0067 


0.0042 


1.5800 


0.1140 


Missing data on high school GPA 


0.1558 


0.0753 


2.0690 


0.0380 


Missing data on developmental courses 


0.0056 


0.0797 


0.0710 


0.9440 


Missing data on first generation 


-0.0854 


0.0505 


-1.6900 


0.0910 


Missing data on income 


0.0341 


0.0357 


0.9540 


0.3400 


Missing data on finishing tasks 


-0.0764 


0.0682 


-1.1200 


0.2630 


Missing data on B32 


-0.0618 


0.0679 


-0.9110 


0.3630 


Missing data on services for limited-English 


-0.2639 


0.1734 


-1.5220 


0.1280 


Missing data on student orientation 


0.1009 


0.1358 


0.7430 


0.4570 


Missing data on reentrance counseling 


0.0212 


0.0684 


0.3100 


0.7570 


Missing data on classroom instr.— basic skills 


0.1128 


0.1686 


0.6690 


0.5030 


Missing data on tutoring in 1991-92 


-0.0747 


0.0695 


-1.0760 


0.2820 


Missing data on tutoring in years 4-6 


0.1830 


0.1451 


1.2610 


0.2080 


Missing data on counseling in years 4-6 


-0.1256 


0.0818 


-1.5370 


0.1240 


Missing data on services for disabled in years 4-6 


-0.2264 


0.1616 


-1.4010 


0.1610 


Missing data on field trip in years 4-6 

Missing data on contact with support services in yrs. 


0.3364 


0.2092 


1.6080 


0.1080 


4-6 


-0.2524 


0.2092 


-1.2060 


0.2280 



SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 
(SSS): Examination of Student Outcomes After Six Years, 2010. 
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Appendix B: Methodology 



Highlights 



• This study examined two types of outcomes to evaluate the effects of SSS. These were 
students’ academic performance (i.e., college grade point averages and total number of credits 
earned), and retention in higher education and degree attainment (including transfers from 
two-year to four-year institutions). 

• The study used transcript data to track the academic performance of about 2,900 freshman SSS 
participants over six years. Propensity scoring was used to select a comparison group of about 
2,900 students with similar demographic characteristics who also were tracked over six years. 

• Researchers collected detailed information about students’ participation in SSS, including the 
number of hours of services received in each of nine categories: instructional courses, 
professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, 
cultural events, and services for the disabled. Both SSS participants and the comparison group 
also provided self-reports about services they received. 

• Because students in the comparison group were still relatively advantaged compared with the 
SSS participants, additional statistical adjustments were used to correct for these differences. 
The variables chosen for these adjustments include students’ demographic characteristics, 
academic background, and attitudes, and institutional characteristics of the schools they 
attended. 

• Using regression analysis, the study developed statistical models to examine the relationship 
between the SSS services received and the measures of effect. 

• Regression coefficients were used to calculate the effects of SSS for each student based on the 
number of hours of services that each received. Summary statistics then were developed to 
describe the average effect of individual SSS services and of SSS overall. 
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Introduction 



This section provides a description of the study design and of the issues that were 
involved in the data analysis. Although there is some overlap in topics, it does not attempt to 
repeat all of the discussion provided in chapter 2 on the study design. Chapter 2 is intended to 
provide the general background that is needed to understand the structure of the report, while this 
section provides additional historical information about the development of the study and of the 
statistical models, along with some additional details on the topics discussed in chapter 2. (There 
necessarily also is some discussion of the research methodology when interpreting the statistical 
models used for the evaluation, especially in chapter 5, which weighs the various statistical models 
and attempts to determine whether some analytic approaches are more effective or trustworthy 
than others. Again, this section does not attempt the repeat that analysis but focuses on more 
general issues concerning the study design.) 

While this report adds new methodological approaches that have not been used in 
earlier reports (i.e., the use of propensity scores within the statistical models, and the use of 
hierarchical linear modeling), in many ways the form of the statistical models still builds upon 
research that was conducted in earlier phases of the study. We did not attempt to reinvestigate 
every methodological choice that had been examined in the past. Thus, some parts of this 
appendix refer to earlier investigations (e.g., comparing the value of various approaches when 
investigating the effects of SSS on first-year GPAs) and do not attempt to update the analysis 
using the final data set (e.g., looking at six-year cumulative GPAs). 

Response Rates 

Following is a list of the different data sources used in preparing this analysis, 
together with the dates the data were collected and the response rates. 



Data source 


Dates of data collection 


Response rate 


Lists of all SSS participants 


1991-92 


28 of 30 sites 


Freshman files for all freshmen in each 
SSS and non-SSS study site 


1991-92 


28 of 30 SSS sites; 19 of 20 non-SSS 
sites 


First-year baseline student survey 


1991-92 


86 percent 


Service records of SSS participants 


1991-92 


86 percent 


Follow-up student survey 


1994-95 


86 percent 


Student transcripts 


1994 


97 percent from initial 47 institutions; 92 
percent from additional institutions 


Follow-up student survey 


1997-98 


77 percent 


Student transcripts 


1997-98 


92 percent of schools; 97 percent of 
transcripts 



Longitudinal Analysis 

This study was designed as a longitudinal analysis of 2,900 SSS participants chosen 
when they first entered college and then tracked over time. Students were regularly contacted to 
obtain information about their attitudes, characteristics, academic progress, and employment, and 
their academic transcripts were collected and summarized in order to measure their academic 
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performance. At the time of this report, six years of data about the students were available. For 
this report, students’ performance is measured in the following ways: 

• Academic performance 

■ GPAs earned in college each year and over all six years combined 

■ Number of credits earned each year and over all six years 

• Retention and degree attainment 

■ Retention at the same institution to the second, third, and sixth years 

■ Retention at any institution of higher education in the third year and sixth 
year 

■ Transfer from two-year to four-year institutions 

For several reasons, it was determined that a longitudinal analysis of college 
freshmen would be the most effective means of measuring the short-term and long-term effects of 
SSS participation. If students were sampled at a later point in their college enrollment, then little 
information would be available about their first years in college, except for information recorded 
on academic transcripts or that students are able to remember and report. Also, because many 
students drop out of college before their second and third years, and because promoting retention 
in college is one of the major goals of SSS, a sample of non-freshmen potentially would be 
statistically biased by excluding those students who were not retained. Finally, an examination of 
SSS programs revealed that SSS services are primarily provided in the freshman year, and that 
there is little difference in the receipt of services after the freshman year; by selecting students as 
freshmen, detailed information could be collected about their participation in SSS services, which 
forms the primary basis for the analysis that follows. 

Depending on the effect measure used, the effects are examined in individual years 
and cumulatively across all six years. Generally, given that SSS participation is typically greatest 
during the freshman year, one might expect that the greatest effect of SSS would also be found at 
that time, though it is also possible that some effects (perhaps especially for retention) may take 
more time to accumulate. On the other hand, since SSS has specific long-term goals such as 
increasing degree completion and transfers to four-year institutions, it is appropriate to evaluate 
SSS in terms of such goals. By looking at each year individually as well as at all years in 
combination, both possibilities can be examined. Moreover, this approach also allows the 
examination of the persistence of benefits of SSS participation: if SSS does improve students’ 
academic performance, is that effect limited to the specific courses and time period when those 
services are received, or are the students learning knowledge and study skills that will help their 
performance in later years as well? 



Quasi-Experimental Design 

Many studies are handicapped by the lack of adequate statistical controls, so that 
when program effects appear to be identified, there may be a strong possibility of other 
confounding explanations for any differences that were found. For example, some possible 
confounding factors are maturation, high motivation to succeed, and differing levels of academic 
skills and knowledge. To limit the danger of such confounding factors, it is best to study two 
roughly equivalent groups, one subjected to the program treatment and the other which is not. By 
comparing the outcomes of the two groups, one can estimate whether the program treatment was 
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associated with improved performance. For this approach to work, it is especially important that 
the two groups be equivalent, and the process for choosing the groups can be critical to the success 
of the study. If people are assigned to the groups on a random basis, then the design is considered 
a true experiment. If exposures to the treatment are not randomized, then the design is quasi- 
experimental. 

Selection of an appropriate comparison group is especially important for an 
evaluation of programs for disadvantaged students. Disadvantaged students are less likely both to 
attend college and to graduate from college. A study that compared SSS students to a typical 
mixture of students would be likely to show negative correlations between participation in SSS 
and student outcomes, not because SSS participation was counterproductive, but because SSS 
students could be expected to have worse outcomes than other students on average. 

Random assignment is often considered the “gold standard” for evaluation studies. 
The random assignment of people to both groups is helpful because there is always the possibility 
that some factor might make one group systematically different from another, and thus explain in 
part any differences that are found. Researchers can consciously try to make the two groups 
equivalent by using some type of matching process, but there is the possibility that the matching 
may not work well (e.g., because data are not available, the data are of poor quality, or researchers 
failed to anticipate all of the characteristics that might be important). The use of randomization 
helps to helps to prevent any systematic differences from appealing between the two groups. 

Despite the strong advantages of random assignment, there are times where it is 
impractical or ineffective. These situations are particularly relevant with regal'd to the SSS 
evaluation. 



■ Failure of randomization to control for critical characteristics. It is 

necessary that the random assignment process be designed to span across all 
important dimensions. Though simple in concept, however, it not always easy 
in practice, and especially if one does not know which dimensions might be 
important. One way that this issue is relevant for the SSS evaluation is that the 
assignment of SSS grants to institutions is itself not random. Institutions must 
choose to apply for the SSS grants, and the grants are awarded in a competitive 
fashion after evaluating the quality of the proposals. It is possible that 
characteristics that affect an institution’s interest and ability to prepare a strong 
proposal might also be important in shaping the institution’ s program design 
and implementation. In this particular case, the potential for bias is not 
necessarily critical as long as the way in which grants are distributed does not 
change, but it may affect the generalizability of the results to new institutions if 
the program is expanded or the selection criteria are altered. 

■ Ethical issues with the use of randomization. Sometimes there are ethical 
reasons why it is difficult to impose randomization. For example, there are 
ethical difficulties with telling a student in the comparison group that he or she 
cannot receive tutoring or some other service from any source while in the 
study. Further, institutions would be likely to consider such a policy as being 
counter to their mission and potentially risky with regard to maintaining 
support and interest from prospective students, parents, alumni, and financial 
supporters. Thus, institutions do often offer supplementary services outside of 



116 




SSS, and needy students often participate in them, making it difficult or 
impossible to select students who have equivalent experiences except for the 
participation in SSS. 

■ Problems in implementation. Sometimes it is difficult to implement a study 
in a way that will make the desired distinction between the treatment and 
control groups. One difficulty with regard to SSS has already been noted: the 
distinction between SSS students and the comparison group is not ideal 
because the comparison students also often received supplemental services. 
Further, even among those students receiving SSS services, SSS is structured 
to allow students freedom in determining which services they receive and how 
much they receive; thus, one cannot force them all to receive equivalent 
amounts of services, and there is a potential that students who choose a 
particular type of service or who choose high levels of services may be 
systematically different from other study participants. SSS students also may 
receive supplemental services from other sources than SSS, and it would be 
both impractical and potentially unethical to prevent the receipt of such 
services (especially since some SSS programs are designed with the 
expectation that the students will also receive supplemental services from other 
sources). 

■ Placebo effects. In medical and other research, it has been well documented 
that the process of being monitored or treated can itself have an effect. 
Researchers try to compensate for this issue in a variety of ways; for example, 
the act of observing a comparison group helps to ensure that all groups have 
the feeling of being monitored, and the distribution of sugar pills (or other 
substitutes) helps to make people feel they are being treated. However, 
depending on the nature and invasiveness of the procedure, it is not always 
possible to hide that one group is being treated differently than another. The 
nature of participation in SSS is such that one cannot disguise whether a 
student is receiving a service, and one could easily hypothesize that the process 
of participating may change a student’ s attachment to the institution or 
motivation to study . 26 

Because of the difficulties with implementing a random assignment approach, this 
study instead used a quasi-experimental approach. The study did not attempt to influence either 
students' inclusion within SSS or their level of participation in supplemental services. It used a 
comparison group of students who were not participants in SSS but were chosen because they had 
similar characteristics to the participants . 27 The method and effectiveness of the selecting the 
comparison students is discussed in greater detail in a following section. The use of such a quasi- 
experimental design had both advantages and disadvantages: 



26 Students often are unaware of the source of the service (e.g., whether it was provided through SSS or some other 
source), but the critical point is that such students are aware that the institution provided special supplemental services 
to them. 

27 A few schools did accept some marginal students for enrollment into the institution only if those students agreed to 
participate in SSS as a condition of their enrollment. The study did not attempt to change institutions" policies in either 
direction, neither forbidding conditional enrollments nor requiring them. 
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■ The study was able to evaluate SSS in the same form as it was actually being 
implemented. If the conditions of SSS had been changed (e.g., by requiring a 
certain level of participation), the nature of the SSS program might be have 
been so radically changed that the study results might no longer be applicable 
toward estimating the effects of SSS. For example, it is possible that students 
select the type and amount of services they receive based on their perceptions 
of the effectiveness of the services (e.g., by discontinuing contacts if a service 
does not seem to be helpful), and that students vary in terms of which service is 
most effective (e.g., based on differences in learning style or academic skills 
and background). Forcing all students to receive uniform experiences might 
not only be difficult to implement (because of student resistance) but might 
obscure the effects of SSS. One difficulty is that students may not be as likely 
to benefit from a service if their attendance is forced, and another is that if 
students do differ in terms of which services will be most helpful, denying the 
possibility of self-selection may water down the estimated effect of each 
service by including students who are less able to benefit. Whatever the 
findings, it would be difficult to extrapolate those findings to SSS programs as 
they are actually implemented, when students can determine for themselves the 
amounts and types of services that they receive. The actual effect might be 
either higher (because of students’ self-selection into those services that are 
most helpful) or lower (because of lower participation levels) than those 
observed under controlled conditions. 

■ The fact that students’ receipt of services was not controlled or randomized 
does create a potential for bias, if the factors that are associated with 
participation are also related to student outcomes. For example, if those 
students who participate most are highly motivated to succeed, that motivation 
might help the students to be successful regardless of what supplemental 
services they receive. Conversely, if the students who participate the most are 
the neediest students (who seek more help because they need more help), then 
their outcomes may tend to be worse because of their greater need. Note that 
this issue is very different from the one raised in the first bullet. If students’ 
self-selection is associated with whether the services are helpful, then it is 
critical to allow such self-selection to properly estimate program effects. On 
the other hand, if students’ self-selection is associated with other factors that 
may affect student outcomes, then it may be difficult or even impossible to 
separate the effect of the service from the “effect” of the student 
characteristics. This issue is discussed in greater detail later within this 
chapter. 



The Comparison Group of Non-SSS Students 

In order to measure the effects of SSS on students’ academic performance, one needs 
some basis for knowing how the students would have performed if they had not been in SSS. It is 
not appropriate to compare SSS students to typical college students because the academic or 
economic disadvantages that are required for SSS eligibility are known to be negatively associated 
with students’ academic performance. Depending on the degree to which SSS helps participants 
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to overcome their disadvantages, they might be helped through their SSS participation without 
necessarily performing as well as “typical” students. 



Method of Creation of Comparison Group 

The study design included the selection of a comparison group of freshman students 
who were chosen to be as similar as possible to the SSS participants. The comparison group was 
chosen by using regression analysis to calculate propensity scores of students' likelihood of 
participation in SSS based on demographic data that were available from the colleges. 28 The 
derived formulas were then used to choose at each institution a comparison group of students 
whose propensity scores showed similar distributions. To lessen the risk that an institution might 
not have non-SSS students who were comparable to the SSS participants, the study also used 
propensity scores to select a comparison group of students from a similar institution that had no 
SSS programs. In this way, even if the SSS programs "skimmed" the most disadvantaged 
students, the study would include some comparison students for whom no such skimming was 
possible. 29 



No constraints were placed on the comparison group in terms of whether they 
participated in other non-SSS services, because if the SSS students had not participated in SSS, 
one might expect that they would have received many of the same non-SSS services. However, 
both the SSS participants and the comparison group were asked in the student questionnaires to 
describe all of the services that they received so that the effects of these services could be 
measured. 



Comparison With SSS Students 

As the 1997 report documented, while the propensity scoring was effective in 
selecting students who were more like the SSS participants than typical students, the differences 
were only partially overcome. In particular, SSS students tended to be more disadvantaged than 
the students in the comparison group. The primary reason the SSS students and the comparison 
group were not more similar appears to be the lack of adequate data for fully comparing the two 
groups of students. For example, some institutions did not have information about the race or 
ethnicity of their students or did not have information about the students’ finances, especially for 
those who were not receiving assistance. The fewer the items that were available, the less 
powerful were the propensity models that were developed; most institutions were able to supply 
only a limited amount of data. Another reason that it was sometimes difficult to select comparable 
non-SSS students is that at a few institutions, case studies revealed that the SSS programs were so 
highly targeted that there were no comparable non-SSS students with similar characteristics. The 
number of SSS students at such institutions, however, was sufficiently small as to have only a 



2S The formulas that were used for calculating propensity scores varied from one institution to another because 
institutions differed in the amount of information that they were able to provide for use in the statistical model. 

29 However, some institutions only admitted certain types of students if they participated in SSS. These institutions by 
definition did not have comparable students outside of SSS. To the degree that the matching non-SSS institutions had 
similar admissions criteria, even those institutions might not have students with comparable disadvantages because 
without an SSS program to compensate for the students' deficiencies, those institutions might not accept such students 
on even a conditional basis. 
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minor effect on the overall averages. Also, the students in the highly targeted programs were 
actually less likely to be disadvantaged than the other SSS students (e.g., 77 percent were white, 
compared with 35 percent of other SSS students) so that targeting was not the major source of the 
differences between the SSS and comparison groups. Whatever the cause, there were systematic 
differences between the SSS participants and the comparison group, with the SSS participants 
being more disadvantaged. The analysis therefore required the use of statistical adjustments to 
correct for these differences (discussed below). 

The 1997 report also shows that there generally were only small differences between 
the comparison students based on whether they were at SSS or non-SSS institutions. Because 
statistical adjustments for the differences between SSS and non-SSS students are required in any 
case, and these adjustments can also compensate for small differences between the two types of 
institutions, the distinction between the two comparison groups is unnecessary. Accordingly, this 
report combines both groups of comparison students into a single group for most analyses. 



Use of Statistical Adjustments 

Because the comparison group still might be anticipated to outperform the SSS 
students on academic measures, additional statistical adjustments were necessary. The use of 
statistical adjustments also was necessary for another reason: even among the pool of SSS 
participants, there was substantial variation in student characteristics that might reasonably be 
expected to affect the level of services that students use. For example, a student's high use of 
tutoring or counseling might reflect a high level of academic need so that the use of SSS services 
could be negatively correlated with measures of academic performance. 

In order to adjust for these differences in student characteristics, two steps were 
taken. First, the analysis pooled the student data across all of the participating institutions; in this 
way, even if comparable non-SSS students could not always be found at the same institution as a 
particular group of SSS students, students with similar characteristics could often be found at one 
or more of the other institutions. Second, multivariate regression models were used to statistically 
adjust for differences in student characteristics. Because the purpose was not to measure the effect 
of a specific student characteristic, but rather to adjust for the cumulative effect of many 
characteristics, a relatively large number of variables were used in the regression models. 
Specifically, a variable was included in the model if it was of theoretical interest (e.g., measures of 
SSS participation and eligibility) or it proved statistically significant at the 0.05 level. 

Because the intention of the model was to statistically adjust for factors that might be 
related both to SSS participation and to students' GPAs or retention, a relatively comprehensive 
list of variables was included in the model. If the intention was to measure the specific effects of 
one of these factors, such a procedure might result in biased estimates. For this particular study, 
however, the goal of providing a complete adjustment was deemed more important. 
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Measurement Issues 



Variability of SSS Services 

One theoretical decision that has important implications for evaluating SSS is 
whether to treat SSS as a relatively uniform intervention into students' lives or as a set of services 
that varies tremendously from one institution to another, and from one student to another even 
within an institution. From the standpoint of policy evaluation, there is considerable value in 
stating an overall effect of the SSS program and, thus, treating SSS as being relatively uniform. 
However, both our qualitative case studies and our quantitative data indicate that SSS services are 
not received in a uniform manner. 

• SSS programs differed in the way they were organized. Some primarily focused 
on providing a single service, some were the only (or at least primary) provider of 
all support services at the institution, and some provided a home base on campus 
that served the “whole student” by providing a broader range of services to 
facilitate the students’ integration on campus and by seeing that any needed 
supplemental services were provided. Because only three programs fit the “all 
service” category, and because the case studies suggested that the home-based 
programs deserved the closest analysis, the analysis in this study contrasts home- 
based programs with all other programs. 

• SSS programs also differed in the extent to which the SSS services were blended 
with other services on campus. The analysis in the report uses the degree of 
blending as another way of differentiating between programs. 

• SSS programs differed greatly in the types and amount of services they provided. 
For example, some programs provided separate instructional courses (e.g., 
developmental courses) that were offered exclusively to SSS students, while 
others offered similar courses that made no distinction between SSS and non-SSS 
students. Exhibit B-l shows the percentage of SSS programs that provided each 
of nine SSS services to at least one freshman student in the sample. 30 While 
some services, such as professional counseling and peer tutoring, were almost 
universally available to students (with over 90 percent of the students at programs 
that provided the service), four of the nine services were offered by less than half 
of the programs. 

• SSS students differed greatly in their use of services even at a single institution. 
This is most obviously true of services offered to the handicapped, because 
relatively few students were eligible for such services, but it was also true of 
more general services such as peer tutoring: some SSS students received no 
services, others received only small amounts, and still others received large 
amounts. No service was used by all students for whom the service theoretically 
was available. For example, while 96 percent of all programs offered 



30 In some programs, a service may have been available, even though no sampled freshman had received the service. For 
example, the service might be offered only to nonfreshmen, or it might be such a specialized service (e.g., for 
handicapped students) that no sampled students used it during the study. The definition used here requires specifically 
that the service be available to the SSS freshmen in the sample, as measured by the fact that at least one such freshman 
received the service. Additional information about the nine services is provided in exhibit 1-1 in chapter 1. 
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professional counseling to freshmen, only 82 percent of all the sampled freshmen 
in those programs received professional counseling. 31 In fact, only two 
services — professional counseling (82 percent) and workshops (62 percent) — 
were received by a majority of the students to whom they were available, while 
five of the nine services were received by less than a third of the students that 
might have received them. And only one service (professional counseling at 80 
percent) was received by a majority of all SSS students. 



Exhibit B-l. Percent of students in SSS programs in 1991-92 that provided each of nine 
SSS services 




0 10 20 30 40 50 60 70 80 90 100 

Percent of SSS students at an institution offering the 

SOURCE: U.S. Department of Education, Planning and Evaluation Service (now known as the Policy and Program Studies Service), 
National Study of Student Support Services (SSS), Service Record Analysis, 1991-92. 



Because of this great variation in the availability and use of the SSS services, a 
measure of the effects of SSS can be misleading unless the variation is taken into account. The 
group of SSS students includes many who received very few services through SSS and thus who 
were not very different from the comparison group in their educational experiences. Also, if some 
SSS services are more effective than others (as suggested later in this report), then even students 
who appeared to receive many SSS services may not have received the services that were most 
valuable, so that their educational experiences again may not be meaningfully different from those 
of the comparison group. 

An implication of these differences is that it is difficult to describe SSS in terms of 
the "average" effects. Because there are many students who participate in SSS in only a minimal 
way (9 percent had only one service contact over the entire freshman year, 7 percent received less 
than one hour of total services in the first year, and 22 percent received between one and five 
hours), the average tends to understate the effect of SSS on those with whom it is most involved. 



31 A slightly lower number — 80 percent — received professional counseling if one also includes SSS students who 
attended colleges that did not offer professional counseling. Typically, for services that were less widely offered than 
professional counseling, the gap between the two percentages is much greater. 
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The financial costs connected with SSS are most associated with students who are heavy users of 
the services, while the average effect would instead give a substantial weight to students who 
neither required many resources nor would be expected to feel much benefit. On the other hand, 
excluding those students who barely participated in SSS could also lead to misinterpretations of 
the data. If the reason that those students showed minimal participation is because they perceived 
that SSS was not helping them, then excluding them would be biasing the study results by 
excluding the “failures” (i.e., those who the program failed to help, regardless of how they did on 
the various outcome measures). 

This analysis therefore uses multiple approaches. Some statistical models 
differentiate between SSS students based on the types of programs in which they were involved 
(home -based versus other, and blended versus separate), the types of services they received, and 
the number of hours of services they received. As an alternative, we also examined SSS 
participation through a single dichotomous measure that was based only on whether students 
participated in SSS during their freshman year, and that did not take account of either the type or 
amount of services they received. We also considered alternative means of measuring SSS 
participation, such as summing each student's participation across all nine services to create a 
single total for each student. This approach has the advantage of simplifying the measurement of 
SSS participation while still allowing for variations in the use of services. However, such an 
approach assumes that an hour of one type of service is equivalent to an hour of another, and tends 
to especially place a high weight on those services that consumed the most time (especially on 
development courses that were set aside only for SSS students). Both assumptions might be 
considered questionable. 

One might speculate that the presence of certain types of SSS services provides 
important information about the general characteristics of the SSS programs. For example, only 
some SSS programs offered participation in cultural events as part of their SSS program, which 
might be an indicator that the program was sensitive to students’ integration on the campus as well 
as to academic issues. Thus, it is possible that the effect measures used here might sometimes 
measure more general program characteristics than the specific services noted here. For example, 
it is possible that participation in cultural events by itself has no effect on student outcomes, but 
that rather it is the orientation of the program (which is measured through the offering of cultural 
events) that is the key factor. Possibly the “service” presenting the greatest such risk is the 
measure of home-based programs, because the presence of such a home base on campus helps to 
assure that students receive a coordinated package of benefits; thus, this measure is not really a 
measure of a particular service but of a set of services. The measure of blended services could 
present similar risks. Except for these two measures, however, the wide variation in the receipt of 
SSS services even within individual institutions lessens the likelihood that the measures are only 
serving as surrogates for more general program characteristics. It is certainly possible that other 
services might provide some of the same effects (or even increased effects) as the services 
examined here. In fact, this would be true regardless of whether the measures are surrogates for 
more general program characteristics. Nevertheless, it seems reasonable to conclude that the 
effects are still the result of participation in particular services rather than being more general 
indicators of program effects. 
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Measurement of SSS and Other Student Services 



The measures of first-year SSS services were based on service records provided by 
the cooperating institutions that indicated the number of minutes (here converted to hours) of 
students' participation in each service. Data were collected for nine different categories of 
services: instructional courses, professional tutoring, peer tutoring, professional counseling, peer 
counseling, labs, workshops, cultural events, and services to the disabled. Additionally, data were 
collected using a number of subcategories of these services; for example, the category of peer 
tutoring was subdivided into general peer tutoring and tutoring in each of five different specific 
subject areas (English, mathematics, science, social science, and general). These subcategories 
and the percentage of SSS students using each type of service are presented in exhibit 1-1 in 
chapter 1. Some analyses were conducted using selected subcategories to determine whether the 
general categories or subcategories were more useful. The general categories are used in this 
report for simplicity and because the subcategories provided no useful additional information; 
generally the main effect of using the subcategories was to reduce the statistical significance of the 
findings, probably because of the reduced number of students getting a service when such detailed 
subcategories were used. 

In order to reduce the burden the participating institutions, and because the freshman 
year was viewed as the primary year in which students participate in SSS, the study only obtained 
service records for the first year (1991-92). For services received in later years, only students’ 
self-reports were used. Students were not asked to indicate the source of the services (they would 
not necessarily know the source, especially since many SSS projects adopt unique names at 
individual schools rather than using the SSS name), so these data do not distinguish between SSS 
and non-SSS services. 

In order to properly measure the effects of the SSS services, it was judged necessary 
to also measure students' use of non-SSS services because SSS programs often referred students to 
non-SSS services, and because students often could receive equivalent services without 
participating in SSS. For example, if two students had equivalent abilities and backgrounds and 
one student benefited from SSS services while the other benefited from equivalent non-SSS 
services, an analysis that excluded the non-SSS services might falsely conclude that SSS had no 
effect because there was no measurable difference in outcomes between the two students. In fact, 
they each may have benefited compared with how they would have performed with no services. 
Similarly, because SSS students may receive non-SSS services, even an analysis that was limited 
to SSS participants would require the measurement of non-SSS services. Otherwise, one SSS 
student could benefit from an SSS service while another SSS student benefited from an equivalent 
non-SSS service, and the SSS service would falsely appear to show no effect because there would 
be no difference in the effect on outcomes. 

Unfortunately, it was not possible to collect data about non-SSS services with the 
same precision as for SSS services, partly because of the wide range of sources of services. 

Instead, this study used students' self-reports of the services they had received, which was typically 
a dichotomous (yes or no) measure. (An exception is for tutoring and counseling, in which the 
students gave more detailed categorical responses — e.g., weekly, monthly — that were converted to 
numeric estimates.) The dichotomous measures were not expected to perform as well as the more 
precise measures of the SSS services (because they lumped together students receiving high and 
low amounts of the services), and an analysis confirmed that they did not. For example, the 
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measure of non-SSS tutoring showed statistically significant effects on first-year GPA when used 
as a continuous measure but not when it was recoded as a dichotomous variable. Thus, this 
analysis may understate the value of non-SSS services. Nevertheless, the statistical results for the 
non-SSS services were consistent with those for the SSS services, with the same types of services 
often being statistically significant for both SSS and non-SSS students. 

The primary difference between the two groups of measures is that in a few cases the 
non-SSS measures were positive and statistically significant when the comparable SSS measures 
were not: for example, for two outcomes cultural enrichment activities were statistically 
significant when the comparable SSS measure was not, and measures of counseling often 
performed better than the measure of first-year SSS counseling. These small and occasional 
differences might be attributed to extraneous factors, such as differences in the number of students 
receiving services, or the use of targeted SSS counseling to help students facing academic 
difficulties. Given the consistency of the findings for SSS and non-SSS services, it does not seem 
that the differences in metrics had a harmful effect. Further, the primary goal of including the 
measures of non-SSS services was not to precisely measure their effects but to obtain an overall 
measure of the mean effect of each service so that the regression intercept (in combination with the 
other demographic variables) reflected the estimated result if a student received no services; this 
approach allowed a better estimate of whether SSS students were helped. 

The measurement issue is further complicated by the fact that students' self-reports 
did not indicate whether the services they received were provided through SSS or through some 
other mechanism. One alternative was to leave the students' self-reports unadjusted, which would 
risk double-counting an SSS service as being provided both through SSS and also outside of SSS, 
while another alternative was to assume that whenever a student received an SSS service, the 
student’s self-report must be referring to that SSS service and not to some alternative source. The 
case studies suggested that it is rare for a student to receive the same service through both SSS and 
outside of SSS; rather, when students receive both SSS and non-SSS services, the non-SSS 
services are typically in areas in which no comparable SSS services are provided at the institution. 
Thus, the second alternative was judged the most reasonable, and student self-reports were 
adjusted to eliminate any double-counting of services . 32 



The Use of Multiple Measures of SSS Services 

By using multiple measures of SSS services, there is a risk that if two services were 
highly correlated, their joint inclusion might result in increased standard errors or a 
mismeasurement of the relative effects of the services. To check whether the use of multiple 
measures simultaneously had an effect on the results, each of the nine measures of SSS services 
was also run separately in regressions to estimate the effects of SSS on students’ first-year GPAs. 
This procedure had no effect on which variables showed positive and statistically significant 
results; the only important difference was that one variable (professional counseling) that 



32 One reviewer had the concern that the combination of using a highly precise measure of SSS services with a 
dichotomous measure of non-SSS services would result in the effects of non-SSS services being absorbed by the more 
precise SSS measures. However, the strategy that was used to avoid double-counting helps to prevent this problem 
from occurring, because only one measure of a particular type of service can be nonzero for any given student. There 
is a risk that the measurement of the SSS effects will be overstated if a student received both an SSS service and an 
equivalent non-SSS service, but the likelihood of such an occurrence is so low that this is not a serious issue. 



125 




otherwise showed negative and statistically significant results was still negative but no longer 
statistically significant. 

There are probably two explanations for this. First, because SSS overall had a 
positive effect and no other variable was included in the equation to measure that effect, the 
variable for SSS professional counseling picked up some of that positive effect, canceling out 
some of the negative association between the need for counseling and students' GPAs. Second, 
when professional counseling alone was in the model, the remaining variables in the regression 
equation were better able to adjust for the differences between students who received counseling 
and those who did not, while when all nine SSS services were included together, the coefficients 
that were appropriate for most SSS services were not as appropriate for professional counseling. 
The primary change was in the intercept, suggesting that students who received professional 
counseling started with some type of disadvantage relative to other SSS students that could not be 
captured through the other measures of student characteristics. Though the negative coefficient for 
professional counseling is undesirable (in the sense that some aspect of students' performance is 
not being properly explained), this finding best supports the conclusion that a more complex 
model is needed that more fully captures the differences between students who received 
counseling and those who did not, not that there is a problem with including individual measures 
of each SSS service. 



Measures of Student Outcomes 

Several measures of student outcomes were developed for this report: students’ 
GPAs, the total number of credits earned, students' retention at the same college or within any 
college, students’ retention or completion of a baccalaureate degree, and students’ transfer from 
two-year to four-year institutions. Following is a more detailed description of the derivation of 
these measures. 

Students’ GPAs were calculated for each year individually and for all six years 
combined using the transcript data. In order to standardize the grading system, all grades were 
converted to a standard four-point scale, with an increment of 0.3 used for pluses and minuses. 33 
The converted numeric score was multiplied by the number of credits attempted in order to 
compute a weighted GPA. 

The number of credits earned was based only on courses taken for regular credit. 
Institutions sometimes differed in the way that credits were assigned, with developmental courses 
counting for regular credit at some institutions and not at others. Each institution’s definition was 
accepted, without attempting to convert all credits to a similar coding scheme. For those colleges 
that divided the academic year into three terms (rather than two semesters), the credits earned were 
multiplied by two-thirds in order to create a standardized semester-based measure. 

Retention at the same institution was measured by the presence of a GPA for that 
year at the sampled institution, without setting a minimum number of courses. Thus, part-time 
students were counted as retained, as were students who enrolled only for a single term in the 
academic year. Retention was measured through students’ self-reports, except that students were 



33 E.g„ “A”=4.0 and “C+”=2.3. 
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counted as retained if their transcript data indicated they were enrolled, even if the students 
reported they were not enrolled or the students did not complete the student questionnaire. 34 

Retention by itself could be a misleading outcome measure since a student who had 
graduated with a baccalaureate degree would typically no longer be in college (unless he or she 
was obtaining additional schooling for a teaching certificate or for graduate education). At the 
same time, many students had not had sufficient time to complete a baccalaureate degree. Thus, a 
combined measure was created that indicated those students who either were still in college or who 
had completed a baccalaureate degree or higher. The study also included separate outcome 
measures based only on what degrees the students had earned (without looking at retention); these 
measures sometimes might be premature (in the sense that some students may not have had 
sufficient time to complete their degrees), but they do not present the same difficulty that a 
retention-only measure would present (i.e., that a success — completing a degree — might be 
interpreted as a failure — no longer being enrolled in higher education). 

The measure of students’ transfers was limited to those students who started out at 
two-year institutions and whose transcripts showed that they had at some time attended a four-year 
institution. The level of the institution was obtained from the Integrated Postsecondary Education 
Data System (IPEDS). 

Missing Data 

In general, if a questionnaire item had substantial missing data, then a two-step 
procedure was followed: the missing values were set to 0, and a new dummy variable was set to 1 
if there had been missing values, and 0 otherwise. In this way, the cases could be retained in the 
analysis, while the dummy variable summarizes the degree to which the cases with missing data 
differed from the overall mean. 

A special situation occurred when structural equations models were used to estimate 
the effect of SSS on retention to the third year. Second-year GPAs were missing for roughly 30 
percent of the students (i.e., because the student at least temporarily left higher education or 
transferred, and the student either did not provide any information or did not provide sufficient 
information for the transfer grades to be obtained), but the previous solution could not be used 
because the second-year GPA was used as a dependent variable in one of the equations. Rather 
than remove these students from the analysis, second-year GPAs were instead imputed by first 
introducing the first-year GPAs, and then iteratively using regression equations to calculate an 
estimated second-year GPA. Without the use of these additional data, the structural equations 



34 There would be some risk of bias if retention rates were calculated through this mechanism, because students who 
remained at the same institution would be counted as enrolled, but students who transferred to another institution might 
not be detected (without the student's report that he or she had attended another college, the transcript would not have 
been collected). Also, the imputation was only in one direction; no students were coded as not enrolled if we lacked 
transcript records to confirm the enrollment. However, the focus of the regression analysis was not to calculate 
retention rates but to calculate the incremental effect of SSS on retention. Assuming that the effects of SSS did not 
vary greatly from one student to another (except based on the differing amounts of services that each received, which 
was included in the model), this recoding of data was helpful by increasing the accuracy of the data and increasing the 
number of cases available for analysis. 
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model could not be estimated. 35 The results from the structural equations models were not greatly 
different from logistic regression models that used high school GPAs in place of college GPAs as 
a measure of academic strength, so unlike the third-year report, this report uses the logistic 
regression models without any imputation of data both for the third-year and sixth-year measures 
of retention. 



Development of the Statistical Models 

In this section, we describe the four types of statistical models that were used in the 
outcomes analysis. Additional discussion of the modeling process, including some alternative 
models that were considered, is provided in the 1997 report. A later section discusses how the 
models were interpreted. 



Choice of Statistical Techniques 

At times, relatively complex statistical techniques were required to solve some of the 
methodological issues that appeared in this study. Because of the possibility that the choice of a 
particular methodological approach might be responsible for the research findings, the 1997 report 
examined each student outcome from multiple statistical perspectives — at the least, by using 
multiple regression to look both for overall effects and for effects associated with particular SSS 
services, but also through using several methods to analyze retention: (1) logistic regression, (2) 
allowing for indirect effects of SSS through its effects on college GPAs, and (3) structural 
equations. In general, the comparison of the different statistical approaches showed that with only 
a few exceptions, the different approaches produced highly similar results, and the techniques that 
appeared the best from theoretical grounds were also the most powerful with regard to their 
predictive power. Thus, rather than repeating the use of multiple perspectives, this analysis builds 
upon the techniques that were identified as best in the 1997 report. This section briefly 
summarizes the findings of the 1997 report, while a more complete discussion of the approach, 
including the results of using the various alternative approaches, is provided in that report. 

One of the key findings from the comparison of multiple approaches was that the 
statistical models were more powerful when each SSS service was treated as a separate variable in 
the overall model than when SSS participation was measured though a simple dichotomous 
measure (i.e., either the student was an SSS participant, or he or she was not). This finding was 
expected because the dichotomous measure mixes together students who were highly involved in 
SSS with students who were barely different in their participation from the non-SSS comparison 
group, and it is not realistic to expect both students to experience the same effects. Further, the 
use of a dichotomous measure ignores how the resources within SSS are distributed; the heavy 
users of SSS are also the ones requiring the greatest resources, so the costs of SSS are largely 
associated with one group, while the effects would be measured by a much larger and very 
different group. This report continues the practice of examining both a dichotomous measure of 



35 However, the results of logistic regression models that omitted second-year GPAs (as an alternative way of handling 
indirect effects of SSS through GPAs) were consistent with the structural equations models, so the results presented 
here do not appear to be an artifact of the imputation process. 
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participation and multiple, continuous measures of participation, so that readers may see the 
importance of the methodological choices involved. 

Second, this analysis repeats the use of logistic regression analysis when the 
dependent variable is dichotomous, because the statistical assumptions for ordinary least squares 
regression analysis are violated when the dependent variable is limited to only two values. 
(However, the HLM models did not use logistic regression because HLM generally was unable to 
converge to a stable estimate. Fortunately, some studies have indicated that OLS regression is 
fairly robust, and tends to produce similar results to logistic regression. The same appeared to be 
true in this study, with the various methodological approaches often producing highly similar 
estimates of effect.) 

In the 1997 report, only the measures of retention required logistic regression 
analysis, but the new measure of transfers to four-year institutions also is dichotomous, so that 
logistic regression analysis is used. In logistic regression analysis, coefficients have a somewhat 
different interpretation than do regression coefficients; they are not as directly interpretable, but 
rather can be transformed to reflect the odds ratio of a student's probability of retention or transfer. 
For example, suppose that a student would normally have a probability of 60 percent of being 
retained until the next year. This probability might be expressed as an odds (60 percent versus 40 
percent, or 60/40 = 1.5). If the logistic regression indicates that the receipt of an SSS service 
might double the odds (i.e., if the odds ratio is 2), then the student's estimated new probability of 
retention (given the receipt of the service) would be 75 percent (i.e., the odds would be doubled 
from 1.5 to 3, and the probability distribution that would produce odds equal to 3 is 75 percent 
versus 25 percent). Thus, the amount by which students’ probability of retention is increased will 
depend upon the original predicted retention rates; if the students are very unlikely to be retained, 
then even a doubling of the (low) probability will still result in a low number, while if the students 
are equally likely to be retained or not retained, the same odds ratio would be associated with a 
much larger change in the probability of retention. 

To estimate the effects of SSS, one therefore needs to know not only the odds ratio, 
but also at least a rough approximation of the students’ probability of retention. To simplify the 
discussion, this analysis often inserts a base rate into the calculation, so that it will easier to judge 
the magnitude of the change in probability. The base rates that were used are 14 percent for 
retention to the sixth year at the same institution, 42 percent for retention to the sixth year at any 
institutions, 34 percent for retention to the sixth year or the earning of a baccalaureate degree or 
higher at the same institution, 60 percent for retention to the sixth year or the earning of a 
baccalaureate degree or higher at any institutions, 6 percent for retention to the sixth year at the 
same institution at two-year institutions, 30 percent for retention to the sixth year at any 
institutions at two-year institutions, 6 percent for retention to the sixth year or the earning of a 
baccalaureate degree or higher at the same institution at two-year institutions, 34 percent for 
retention to the sixth year or the earning of a baccalaureate degree or higher at any institutions, and 
15 percent for transfers from two-year to four-year institutions. 36 



36 The choice of the base retention rate has no effect on the statistical significance of the findings; it only affects how the 
coefficients are interpreted in terms of the size of the SSS effects. Further, even when interpreting the size of the SSS 
effects, the choice of the base retention rate does not have much effect as long as the alternatives are all relatively close 
to each other, as was the case here. Only a rough approximation is needed, especially given the low odds ratios that 
appeared in this analysis (no higher than 1.5 for the SSS variables). For example, if the odds ratio was 1.5, any base 
retention rate from 0.34 to 0.57 would produce an estimate of a 10 percent increase in retention, and any base retention 
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A third finding was that unless great care was used, introducing students’ college 
GPAs as explanatory variables in the regression analysis had the effect of understating estimates of 
SSS on retention. Because students’ college GPAs were affected by participation in SSS, the 
variables that were intended to measure the effect of SSS actually only measured the residual 
effect after the influence of SSS through students' GPAs was removed, not the total impact of SSS. 
Two different approaches produced similar results: leaving college GPAs out of the regression 
(instead using high school GPAs as a measure of academic strength) and using structural equations 
to calculate a base GPA (i.e., the predicted GPA if students had not participated in SSS) that could 
be used rather than the actual GPA (which included the added impact of SSS) to predict retention. 
Because the results were quite similar and the structural equations approaches added several 
complexities (i.e., the software for structural equations models was not designed for logistic 
regression analysis to compensate for the use of a dichotomous dependent variable, and it was 
necessary to impute college GPAs for many students in order to provide enough cases for stable 
estimates), this report uses logistic regression analysis without the use of college GPAs rather than 
using structural equations models. 



Interpretation of Models 

This section discusses how the summary measures of the effects of SSS were derived, 
and how the occasional appearance of negative but statistically significant regression coefficients 
should be interpreted. 



rate from 0.66 to 0.71 would produce an estimated increase of 8 percent. If the odds ratio was 1.2, any base retention 
rate from 0.43 to 0.53 would produce an estimated increase of 5 percent, and any base retention rate from 0.54 to 0.72 
would produce an estimated increase of 4 percent. The base retention rates were chosen based on the observed 
retention rates of the SSS students and the comparison group. 
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Translation of Coefficients into Estimated Effects 



The regression coefficients for SSS services that appeal - in Appendix A may be 
interpreted as the average change in the student outcome that is associated with each hour of 
service received . 37 However, because SSS students varied greatly in the amount of services 
received, the actual impact of the SSS programs cannot be determined from those statistics alone. 
To provide a better summary measure of the impact of SSS, those coefficients that were positive 
and statistically significant were multiplied by the actual hours of services that each student 
received, and the mean among all students was then calculated . 38 This procedure was conducted 
both for each individual service that was statistically significant and for the sum of all statistically 
significant services that each student received. The latter statistic, labeled “total program effect” 
in this report, can be viewed as a single summary statistic that describes the total impact of the 
SSS program when all services are combined together. 

In previous reports, the above calculation typically was performed for two groups of 
students. First, in order to provide an overall measure of the impact of SSS, the calculation was 
performed over all SSS students in the sample, even if a particular student did not participate in 
any services that showed statistically significant effects. This is the most useful statistic if one 
wishes to describe the total impact of SSS as a national program; however, because many SSS 
students participate in only a minimal way, this statistic does not necessarily describe the impact 
on those students most responsible for using SSS resources. Further, especially since some of the 
statistically significant services were not even available to many SSS students, the overall statistic 
fails to indicate the impact that might be produced if SSS were restructured to make greater use of 
services that are shown to be effective. For these reasons, in previous reports the calculation was 
also performed only for students who received statistically significant services (e.g., only for 
students receiving peer tutoring or, for the measurement of the “overall program effect,” only for 
students receiving at least one statistically significant service). This latter statistic is a better 
measure of the value of a specific service, though it does not show the national impact of the 
service. For example, typically each of the nine SSS services was only offered at some of the SSS 
projects and not at others. If a service is offered, say, at half of the SSS projects, it might be 
accurate to say that the nationwide impact of that service was x percent, but it is also misleading in 
the sense that many SSS students never had the option of receiving that service through SSS. In 
this report, the primary focus is on the overall impact of SSS after summing across all SSS 
services rather than on individual services. Also, in order to be conservative with regal'd to the 
“intent to treat” issue (i.e., that omitting people who did not participate might bias the results by 
omitting failures of the project to help students), it seemed best to include all students . 39 



37 The coefficients for the logistic regressions are interpreted somewhat differently and reflect changes in the odds ratio 
that are associated with each hour of service. Odds ratios are discussed in greater detail earlier in this appendix. 

3S In the case of home-based and blended programs, the coefficient was multiplied by 1 if a student was in that type of 
program, and by 0 if the student was in a different type of program. 

39 However, one could differentiate between institutions that do not offer a particular service and students who do not 
participate when it is offered. A student's choice not to participate might be considered a failure of the service (e.g., if 
the reason is that the student has heard negative reports from other SSS participants), but the lack of availability of the 
service might simply indicate that the institution has chosen to offer the service though some means other than SSS. 
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Negative Regression Coefficients 



For some models, and particularly using the measure of the amount of SSS 
professional counseling received by each student, one or more regression coefficients for the SSS 
services were negative but statistically significant. While it seems reasonable that some SSS 
services might be ineffective and show no relationship to student outcomes, it seems much less 
reasonable that a service was actually detrimental to students (e.g., because of quality control 
procedures to win and maintain the program grants, and because students would not be likely to 
participate unless they perceived a benefit from their participation); a more likely explanation is 
that some unmeasured variable (e.g., some academic deficiency) was responsible for both students 
receiving relatively large amounts of a service and also performing poorly on a measure of student 
outcomes. This, in fact, is a common risk of studies of programs to help disadvantaged students, 
because the disadvantages that make a student eligible for special programs also are related to poor 
academic performance; unless appropriate statistical adjustments are made, a negative relationship 
between participation and achievement may simply mean that a program failed to fully 
compensate for students' disadvantages, even if the participation might have been helpful. It is 
because of this risk that considerable effort was given to selecting a comparison group that had 
similar characteristics to the SSS participants, and for using additional statistical adjustments in the 
regression models. 

If negative regression coefficients appeal - only rarely and there is no systematic 
pattern to their appearance, they might appear by chance given the large number of outcome 
measures used in this report. (While this study uses seven categories of outcome measures, it uses 
12 separate measures across those categories; for example, retention/degree completion is 
measured both overall and also separately for two-year and four-year institutions.) For example, if 
a significance level of 0.05 is used for hypothesis testing, then one out of 20 of the findings might 
be expected to be statistically significant simply by chance. 40 This report will interpret negative 
relationships within the larger context of the findings: to the extent that they appeal - idiosyncratic 
and contradicted by more frequent positive findings, they will be ignored and assumed to reflect 
either the disadvantaged status of the SSS students or random aberrations, while consistent 
patterns will be examined to determine the source of the negative relationships. 

The SSS service variable that most regularly produced negative and statistically 
significant coefficients was professional counseling. Additional analysis of the data indicated that 
there were systematic differences between students who received counseling and those who did 
not. For example, students who received no professional counseling had the strongest prior 
academic backgrounds (i.e., based on their mean SAT and ACT percentiles of 0.43). More 
generally, the strength of the students' prior backgrounds was inversely related to the amount of 
counseling they received (with SAT and ACT percentiles of 0.34 for students receiving up to one 
hour of professional counseling, 0.27 for one to two hours, and 0.24 for two to four hours; an 
exception to this pattern is that the percentile was 0.36 for the 6 percent of students receiving more 
than four hours of professional counseling). Similar patterns appeared for other types of 
counseling besides professional counseling. Students who received no peer counseling had the 



40 Not all of the measures are completely distinct. For example, retention or degree completion at four-year institutions 
is likely to be similar to retention or degree completion overall, and perhaps even to retention or degree completion at 
two-year institutions. Thus, it is probably an overstatement to label these as 12 separate measures. Still, to the degree 
that a finding appears highly idiosyncratic rather than showing a consistent pattern across multiple measures, one 
should be wary of giving too much importance to that finding. 
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strongest prior backgrounds with mean SAT or ACT percentile scores of 0.40, while students who 
received up to one hour of peer counseling had mean percentiles of 0.28, and those who received 
over one hour had mean percentiles of 0.25. For non-SSS counseling, the results were mixed 
depending on the type of counseling received. College reentrance counseling showed a similar 
pattern to those above, with the students who received counseling having weaker backgrounds 
(0.35) than those who did not (0.39). For other non-SSS counseling, however, the pattern 
sometimes held and sometimes did not: students who received one to three sessions had stronger 
backgrounds (0.45) than students with more than three sessions (0.38), but also stronger 
backgrounds than those with no counseling (0.37). 

In principle, it should be possible to statistically adjust for the systematic differences 
in student characteristics so that the effect of counseling can be measured. In practice, however, 
the statistical adjustments that were appropriate for the remaining SSS service variables did not 
completely compensate for the relative disadvantage of students receiving professional counseling. 
Even when the remaining SSS service variables were deleted from a regression to predict students’ 
first-year GPAs (under the assumption that different statistical adjustments are required for 
professional counseling than for other services), the relationship remained negative (though 
statistically insignificant). The primary change among the rest of the variables in the regression 
model was a slightly lower intercept, suggesting that students who need professional counseling 
were somehow academically more disadvantaged, but this disadvantage was not well captured by 
the other explanatory variables. Thus, it appears that the proper modeling of characteristics 
associated with receiving counseling would require data that are not currently available. One 
reason for this finding may be that professional counseling was often administered in a manner 
that was distinct from other SSS services; students who appeared to be in academic trouble were 
often called in for professional counseling, so that the receipt of professional counseling was often 
an indicator of academic distress. 

Given the above difficulties, it seems the best interpretation of negative regression 
coefficients is to group them together with other SSS services that failed to show positive and 
statistically significant results; they all are services for which no benefit to students can be 
statistically proven. This does not mean that services that fail to show positive effects are proven 
to have no effect; that is, a failure to prove that a coefficient is different from 0 does not mean that 
the coefficient is equal to 0. For the particular case of negative regression coefficients, we 
especially have reason to think that the model is not working properly, and thus not to give 
excessive attention to those statistics. In this report, the focus is primarily on those services that 
do show positive and statistically significant results, rather than discussing each SSS service 
individually. 



Another question that was examined was the treatment of negative but statistically 
significant regression coefficients when calculating the total impact of SSS. As described in an 
earlier section, the total impact was estimated by multiplying the positive and statistically 
regression coefficients by the actual number of hours of services received for each student, and 
then calculating the total across all services. The negative but statistically significant coefficients 
should not be considered reliable estimates of the effects of SSS, and in that sense they should not 
be included in the calculations. On the other hand, it might be that the size of the positive and 
statistically significant coefficients is due in part to the presence of negative coefficients, and that 
they would be smaller (with a correspondingly smaller total effect of SSS) if the variables showing 
a negative relationship were not included. To investigate this possibility, the regressions in the 
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third-year report were recalculated with the negative but statistically significant variables 
excluded. Overall, the coefficients that previously were statistically significant continued to be 
statistically significant, with coefficients that were roughly the same as those that were found 
previously. For example, the estimated impact of SSS on students GPAs in the first year would 
generally be 0.01 lower than reported in chapter 2 if this procedure is followed. Thus, the 
presence of negative but statistically significant coefficients does not seem to have had an 
important effect on estimates of the effects of other SSS services or on the calculation of the total 
impact. 



For this report, the sixth-year regressions also were examined through alternative 
models in which the negative but statistically significant variables were excluded. Overall, the 
exclusion of the variables typically had little effect on the regression estimates. The two major 
exceptions are that in one model (the impact of SSS on retaining or completing a baccalaureate 
degree), there was a service (SSS workshops) that stopped being statistically significant when 
negative but statistically significant variables were excluded, and in another model (the impact of 
SSS on whether participants at two-year institutions received an associate's degree or higher), 
there was a service (peer tutoring) that became statistically significant when negative but 
statistically significant variables were excluded. In general, deleting variables is riskier than 
including irrelevant variables because deleting a variable may result in biased estimates, while 
including irrelevant variables affects the standard errors but does not bias the estimates. The 
presence of negative but statistically significant coefficients suggests that there is wide variation in 
the impact or distribution of SSS that is not adequately being measured in the model. Most likely, 
the regression model is too small (e.g., it is not including important information about how the 
services are distributed, as noted with regard to professional counseling) rather than too large. The 
approach used in this report is to focus on the statistics resulting from the full model, while 
footnotes discuss the results of the alternative models in which negative but statistically significant 
variables were excluded. 

Estimating Program Impacts 

The method used to estimate program impact varies slightly depending on the 
particular methodology that is used. When dichotomous measures of SSS participation are used in 
multivariate regressions, the regression coefficient for the measure of SSS participation can be 
directly interpreted as the average change in student outcomes that is associated with participation 
in the program. For example, a coefficient of 0.03 in the analysis of GPAs could be interpreted as 
an average improvement (across all SSS students) of 0.03 using a four-point grading scale, and a 
coefficient of 3.1 in the analysis of total credits earned could be interpreted as an average 
improvement of 3.1 credits. When logistic regression is used, the coefficients are more difficult to 
interpret because they represent a change in the probability that the student will achieve the stated 
outcome. 



More specifically, the statistics shown in the tables are not the actual coefficients but 
the log-odds ratio. The odds ratio expresses the improvement in the probability that is associated 
with participation in SSS. For example, if a student’s probability of achieving the outcome is 60 
percent, that probability could also be expressed as an odds of 1.5 (i.e., 60/(100-60)=60/40=1.5). 
An odds ratio of 2 would imply that participation in SSS is associated with doubling the student’s 
odds (i.e., from 1.5 to 3), which would be equivalent to a probability of 75 percent (i.e., 75/(100- 
75)=75/25=3). For that student, then, the increase in probability of achieving the outcome is 15 
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percent (i.e., 75-60). Note that the amount of change in a student’s probability depends on the 
student’s base level of probability of achieving the outcome if he/she does not participate in SSS. 

If the base probability of achieving the outcome instead was 75 percent, the odds would be 
expressed as 3.0 (75/(100-75)=75/25=3.0), and a doubling of the odds to 6.0 would be equivalent 
to a probability of approximately 85.7 percent (85. 7/(100-85. 7)=85. 7/14. 3—6.0), or an increase of 
about 12 percent. For this report, when a dichotomous measure of SSS participation is used, the 
base probability was calculated by determining what probability level, when combined with the 
estimated log-odds ratio that was obtained, would result in the actual outcome probability as 
observed among the SSS students. 41 

When nondichotomous measures of SSS participation are used, the regression 
coefficients cannot automatically be interpreted as the average impact across all students. Rather, 
the regression coefficient represents the level of improvement that is associated with receiving one 
hour of the specified service. Thus, one way of estimating the average impact is to multiply the 
regression coefficient by the average number of hours of that service that were received across all 
SSS students. If multiple services showed positive and statistically significant results, one also 
needs to sum all such estimates in order to estimate the total SSS impact (rather than just the 
impact of one particular service). When logistic regression is used, one would take a somewhat 
comparable approach, except that one must also conduct the appropriate transformations to switch 
from log-odds ratios to changes in the probability of achieving the specified outcome. However, 
when multiple SSS services show positive and statistically significant results, the log-odds ratios 
cannot simply be added. The estimates in this report were computed by multiplying the logistic 
regression coefficients by the actual number of hours for each student, summing the results across 
all SSS variables for each student, estimating the level of impact per student, and averaging those 
results to produce an overall estimate. 

Data on Two-year Institutions 

Several of the outcome measures used in this report were limited to students who 
were attending two-year institutions at the start of the study. The measure of transfers from two- 
year to four-year institutions was defined only for such students, while the measures of retention 
and degree completion were computed for two-year institutions only, as well as across all 
institutions and (sometimes) for four-year institutions only. However, because only 1,200 of the 
5,055 students in the database started at two-year institutions, the models that were limited to such 
students were less likely to show statistically significant relationships. In fact, not only were the 
supplemental services less likely to be significantly related to student outcomes, but all variables 
in the models tended to perform poorly. The final statistical models generally contained only a 
relatively small number of variables, and several variables were retained in the final models only 
because the selection procedures required that at least one variable be retained in each major 
category of variables, regardless of statistical significance. Thus, statistics on two-year institutions 
are not highly accurate, and readers should be wary of overinterpreting the findings. In particular, 
a failure to find statistically significant relationships may only indicate that the number of cases 



41 This approach is a simplification because it ignores the fact that most SSS students also received non-SSS services in 
the first year, plus SSS or non-SSS services after the first year, and both types of services may be associated with 
improved student outcomes. The estimated change in probability that is associated with SSS participation depends on 
what base probability of achieving the outcome is assumed, and the presence of positive and statistically significant 
results for these other supplemental services would suggest that the base probability should be somewhat lower than if 
one assumes that SSS was the only factor producing improved student outcomes. 
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available was not adequate for identifying statistical relationships unless those relationships were 
quite strong. The small number of cases was especially an issue for examining retention or 
baccalaureate degree completion at the same institution. With only a few exceptions, students 
were not able to earn baccalaureate degrees at two-year institutions, and students at two-year 
institutions were not likely to remain enrolled for six years unless they also transferred to four-year 
institutions. Only a total of 84 students at two-year institutions had either remained enrolled at 
that institution or earned a baccalaureate degree at that institution. Thus, the statistical models for 
this student outcome consistently performed poorly, and few valuables of any type were 
statistically significant. 
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